<html>
<head><meta charset="utf-8"><title>MBE discussion · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html">MBE discussion</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="162934609"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/162934609" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#162934609">(Apr 09 2019 at 18:04)</a>:</h4>
<p><span class="user-mention" data-user-id="153740">@Igor Matuszewski</span> <span class="user-mention" data-user-id="216201">@Edwin Cheng</span> you are both interested in MBE. I wonder if we should shcedule a sync-up time to talk through the high-level design?</p>



<a name="162937270"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/162937270" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#162937270">(Apr 09 2019 at 18:32)</a>:</h4>
<p>Sure, im in GMT + 8</p>



<a name="162937709"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/162937709" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#162937709">(Apr 09 2019 at 18:37)</a>:</h4>
<p>Let's try to schedule some time. I am unfortunately pretty busy this week, only Thursday will work for me: <a href="https://doodle.com/poll/z9ka63mz3xcy9wzr" target="_blank" title="https://doodle.com/poll/z9ka63mz3xcy9wzr">https://doodle.com/poll/z9ka63mz3xcy9wzr</a></p>



<a name="162939117"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/162939117" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#162939117">(Apr 09 2019 at 18:51)</a>:</h4>
<p>Done ! I did not know there is this kind of app for scheduling <span aria-label="joy" class="emoji emoji-1f602" role="img" title="joy">:joy:</span></p>



<a name="162949159"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/162949159" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#162949159">(Apr 09 2019 at 20:54)</a>:</h4>
<p>Done!</p>



<a name="162949643"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/162949643" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#162949643">(Apr 09 2019 at 21:00)</a>:</h4>
<p>Good, added to the calendar: <a href="https://calendar.google.com/calendar/embed?src=6u5rrtce6lrtv07pfi3damgjus%40group.calendar.google.com" target="_blank" title="https://calendar.google.com/calendar/embed?src=6u5rrtce6lrtv07pfi3damgjus%40group.calendar.google.com">https://calendar.google.com/calendar/embed?src=6u5rrtce6lrtv07pfi3damgjus%40group.calendar.google.com</a></p>



<a name="163082699"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163082699" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163082699">(Apr 11 2019 at 09:04)</a>:</h4>
<p><span aria-label="wave" class="emoji emoji-1f44b" role="img" title="wave">:wave:</span></p>



<a name="163082733"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163082733" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163082733">(Apr 11 2019 at 09:05)</a>:</h4>
<p><span aria-label="hi" class="emoji emoji-1f44b" role="img" title="hi">:hi:</span></p>



<a name="163082738"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163082738" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163082738">(Apr 11 2019 at 09:05)</a>:</h4>
<p><span class="user-mention" data-user-id="216201">@Edwin Cheng</span> are you around?</p>



<a name="163082899"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163082899" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163082899">(Apr 11 2019 at 09:07)</a>:</h4>
<p><span aria-label="wave" class="emoji emoji-1f44b" role="img" title="wave">:wave:</span>🏻</p>



<a name="163082911"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163082911" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163082911">(Apr 11 2019 at 09:07)</a>:</h4>
<p>So, the rough agenda for the meeting:</p>
<ul>
<li>discuss high-level constraints/goals around macro exapnsion</li>
<li>discuss what is currently implmenting</li>
<li>figure out the next steps</li>
</ul>



<a name="163083005"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083005" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083005">(Apr 11 2019 at 09:09)</a>:</h4>
<p>I must confess that I don't really know deeply how macros work inside <code>rustc</code>. Reading rustc's code will be a major part of work here :)</p>



<a name="163083014"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083014" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083014">(Apr 11 2019 at 09:09)</a>:</h4>
<p>I'll have a question regarding a high-level goal</p>



<a name="163083019"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083019" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083019">(Apr 11 2019 at 09:09)</a>:</h4>
<p>Sure!</p>



<a name="163083038"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083038" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083038">(Apr 11 2019 at 09:09)</a>:</h4>
<p>do we mean for it to be a part of the "librarification" goal? e.g. do we want it to be as pure as possible?</p>



<a name="163083131"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083131" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083131">(Apr 11 2019 at 09:10)</a>:</h4>
<p>I'm not sure how much existing MBE infra relies on rustc internals like interning and so on but I imagine a very long-term goal would be to create a nice abstract library that rustc can also use? or would we have to rely too much on the parser and whatnot?</p>



<a name="163083142"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083142" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083142">(Apr 11 2019 at 09:10)</a>:</h4>
<p>Good one! I think we shouldn't target it for librarification explicitly: librariifing nameres and expansion itself at the same time is more capacity than we have</p>



<a name="163083172"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083172" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083172">(Apr 11 2019 at 09:11)</a>:</h4>
<p>okay, makes sense</p>



<a name="163083180"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083180" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083180">(Apr 11 2019 at 09:11)</a>:</h4>
<p>And i have another question</p>



<a name="163083182"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083182" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083182">(Apr 11 2019 at 09:11)</a>:</h4>
<p>That is, I feel we should make progress on chalk/nameres/lexer first, and then librarify</p>



<a name="163083187"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083187" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083187">(Apr 11 2019 at 09:11)</a>:</h4>
<p><span class="user-mention" data-user-id="216201">@Edwin Cheng</span> go ahead</p>



<a name="163083193"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083193" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083193">(Apr 11 2019 at 09:11)</a>:</h4>
<p>Sure we referenced how rustc works?</p>



<a name="163083256"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083256" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083256">(Apr 11 2019 at 09:12)</a>:</h4>
<p>like should we follow what rustc does 1-1?</p>



<a name="163083260"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083260" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083260">(Apr 11 2019 at 09:12)</a>:</h4>
<p>Yes</p>



<a name="163083266"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083266" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083266">(Apr 11 2019 at 09:12)</a>:</h4>
<p>I'd say probably not</p>



<a name="163083276"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083276" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083276">(Apr 11 2019 at 09:12)</a>:</h4>
<p>I'd like to see how far we can go, using the purest possible interface, etc</p>



<a name="163083287"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083287" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083287">(Apr 11 2019 at 09:13)</a>:</h4>
<p>at least initially.</p>



<a name="163083296"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083296" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083296">(Apr 11 2019 at 09:13)</a>:</h4>
<p>Okay</p>



<a name="163083298"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083298" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083298">(Apr 11 2019 at 09:13)</a>:</h4>
<p>Closing all the gaps should be done as a part of future librarification</p>



<a name="163083323"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083323" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083323">(Apr 11 2019 at 09:13)</a>:</h4>
<p>On the highest level, what we need is a proof of concept that large-scale macro expansion works in IDEs</p>



<a name="163083375"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083375" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083375">(Apr 11 2019 at 09:14)</a>:</h4>
<p>So we should support common macros with a relatively efficient impl</p>



<a name="163083387"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083387" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083387">(Apr 11 2019 at 09:14)</a>:</h4>
<p>Make sense</p>



<a name="163083401"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083401" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083401">(Apr 11 2019 at 09:14)</a>:</h4>
<p>another high level question - I didn't delve deep into untying the nameres/expansion  but I assume these are still highly interconnected</p>



<a name="163083434"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083434" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083434">(Apr 11 2019 at 09:15)</a>:</h4>
<p><span class="user-mention" data-user-id="153740">@Igor Matuszewski</span> the running hypothesis is that they are not really</p>



<a name="163083443"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083443" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083443">(Apr 11 2019 at 09:15)</a>:</h4>
<p>Like, nameres could have a callback expand_macro(MacroId) -&gt; SetOfNames</p>



<a name="163083499"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083499" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083499">(Apr 11 2019 at 09:16)</a>:</h4>
<p>That is, it doesn't care about actual token trees which are produced, only about the names</p>



<a name="163083522"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083522" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083522">(Apr 11 2019 at 09:16)</a>:</h4>
<p>hm, that actually makes sense</p>



<a name="163083550"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083550" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083550">(Apr 11 2019 at 09:17)</a>:</h4>
<p>The hygiene bit is somewhat in the middle between the two though</p>



<a name="163083556"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083556" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083556">(Apr 11 2019 at 09:17)</a>:</h4>
<p>what was the case where we had feared it needs to go back and forth?</p>



<a name="163083567"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083567" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083567">(Apr 11 2019 at 09:17)</a>:</h4>
<p>About not being incremental enough? Or whcih case?</p>



<a name="163083568"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083568" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083568">(Apr 11 2019 at 09:17)</a>:</h4>
<p>when one expansion expands to a glob use and then another expansion uses a (now) shadowed identifier? can</p>



<a name="163083569"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083569" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083569">(Apr 11 2019 at 09:17)</a>:</h4>
<p>can't remember</p>



<a name="163083578"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083578" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083578">(Apr 11 2019 at 09:17)</a>:</h4>
<p>Ah, time-travel</p>



<a name="163083658"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083658" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083658">(Apr 11 2019 at 09:18)</a>:</h4>
<p>Yeah, the <code>SetOfNames</code> should also include set of (glob) imports, but this seems like a strictly name-res concern</p>



<a name="163083664"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083664" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083664">(Apr 11 2019 at 09:18)</a>:</h4>
<p>I can't remember the details now :( sorry about that</p>



<a name="163083703"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083703" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083703">(Apr 11 2019 at 09:19)</a>:</h4>
<p>okay so I imagine we try to go as far as we can using the purest possible interface and don't think about nameres right now, correct?</p>



<a name="163083714"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083714" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083714">(Apr 11 2019 at 09:19)</a>:</h4>
<p>Correct!</p>



<a name="163083729"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083729" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083729">(Apr 11 2019 at 09:19)</a>:</h4>
<p>Purity of interface is one of my personal high-level design goals here</p>



<a name="163083789"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083789" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083789">(Apr 11 2019 at 09:20)</a>:</h4>
<p>let me describe what I've came up with so-far</p>



<a name="163083818"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083818" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083818">(Apr 11 2019 at 09:20)</a>:</h4>
<p>first, we have mbe and proc macros. I think it would be cool if both used exactly the same interface. Basically, making macro_rules a special-cased proc macro</p>



<a name="163083829"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083829" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083829">(Apr 11 2019 at 09:21)</a>:</h4>
<p>The interface is the ra_tt crate</p>



<a name="163083858"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083858" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083858">(Apr 11 2019 at 09:21)</a>:</h4>
<p>It defines a <code>TokenTree</code>  structure, which is more or less copy-pased from proc_macro2</p>



<a name="163083934"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083934" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083934">(Apr 11 2019 at 09:22)</a>:</h4>
<p>crucially, <code>ra_tt</code> does not depend on any other compiler crate</p>



<a name="163083954"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083954" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083954">(Apr 11 2019 at 09:22)</a>:</h4>
<p>Another bit that is has is that its tokens have identifies (each identifier as an u32 associated with it)</p>



<a name="163083973"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083973" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083973">(Apr 11 2019 at 09:23)</a>:</h4>
<p>I "hope" that TokenIds should be enough to associated arbitrary information (hygiene &amp; spans) with tokens</p>



<a name="163083998"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163083998" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163083998">(Apr 11 2019 at 09:23)</a>:</h4>
<p>That is all for the "pure interface" bit</p>



<a name="163084068"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084068" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084068">(Apr 11 2019 at 09:24)</a>:</h4>
<p>That is, macros consume and produce TokenTrees. The question is, how do we integrate thouse back in the AST?</p>



<a name="163084108"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084108" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084108">(Apr 11 2019 at 09:25)</a>:</h4>
<p>Specifically, we need to convert from AST to tokens, from tokens to AST, and we need to preserve hygiene and span information along the way</p>



<a name="163084122"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084122" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084122">(Apr 11 2019 at 09:25)</a>:</h4>
<p>so each token has an associated u32 with it, which can hold info like hygiene and spans, right?</p>



<a name="163084126"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084126" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084126">(Apr 11 2019 at 09:25)</a>:</h4>
<p><span class="user-mention" data-user-id="153740">@Igor Matuszewski</span> right</p>



<a name="163084130"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084130" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084130">(Apr 11 2019 at 09:25)</a>:</h4>
<p>(to  reiterate)</p>



<a name="163084185"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084185" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084185">(Apr 11 2019 at 09:26)</a>:</h4>
<p>basically, you'll have a <code>hygiene: FxHasMap&lt;u32, ExpansionId&gt;</code></p>



<a name="163084189"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084189" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084189">(Apr 11 2019 at 09:26)</a>:</h4>
<p>that is, a hash-map on the side</p>



<a name="163084200"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084200" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084200">(Apr 11 2019 at 09:26)</a>:</h4>
<p>so we explicitly convert AST to tokens for a macro call?</p>



<a name="163084218"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084218" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084218">(Apr 11 2019 at 09:27)</a>:</h4>
<p>since we have the concrete syntax tree (full-fidelity?) I imagine transforming this to tokens shouldn't be hard</p>



<a name="163084219"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084219" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084219">(Apr 11 2019 at 09:27)</a>:</h4>
<p>yes <a href="https://github.com/rust-analyzer/rust-analyzer/blob/master/crates/ra_mbe/src/syntax_bridge.rs#L18" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/blob/master/crates/ra_mbe/src/syntax_bridge.rs#L18">https://github.com/rust-analyzer/rust-analyzer/blob/master/crates/ra_mbe/src/syntax_bridge.rs#L18</a></p>



<a name="163084282"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084282" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084282">(Apr 11 2019 at 09:28)</a>:</h4>
<p>Yes we already have this</p>



<a name="163084298"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084298" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084298">(Apr 11 2019 at 09:28)</a>:</h4>
<p>Specifically, we walk the tree, prodce tokens (assigning incrementing <em>local</em> ids) and also produce a <code>TokenMap</code> which should help with managing spans, but which is not actually used :(</p>



<a name="163084337"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084337" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084337">(Apr 11 2019 at 09:29)</a>:</h4>
<p>and what is <code>tt::TokenId</code> specifically?</p>



<a name="163084339"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084339" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084339">(Apr 11 2019 at 09:29)</a>:</h4>
<p>The bit about local ids is important: for each macro call, the id of the first token will be zero. This helps with incrementality, b/c the token tree is the same regardless of the position of the macro_call</p>



<a name="163084345"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084345" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084345">(Apr 11 2019 at 09:29)</a>:</h4>
<p><code>pub struct TokenId(pub u32);</code></p>



<a name="163084354"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084354" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084354">(Apr 11 2019 at 09:29)</a>:</h4>
<p>and that's the u32 that's used as the key for hygiene info?</p>



<a name="163084380"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084380" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084380">(Apr 11 2019 at 09:30)</a>:</h4>
<p>yes</p>



<a name="163084433"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084433" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084433">(Apr 11 2019 at 09:30)</a>:</h4>
<p>and also as a key in the <code>TokenId -&gt; TextRange</code> map</p>



<a name="163084437"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084437" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084437">(Apr 11 2019 at 09:30)</a>:</h4>
<p>ah, alrighty</p>



<a name="163084480"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084480" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084480">(Apr 11 2019 at 09:31)</a>:</h4>
<p>Yeah, so, we get an TokenTree. We then run macro black-box, which gives us another token-tree, but with the same (identify-wise) tokens</p>



<a name="163084541"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084541" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084541">(Apr 11 2019 at 09:32)</a>:</h4>
<p>Which we feed back into the parser in token_tree_to_ast_item_list, and get the SyntaxTree back</p>



<a name="163084566"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084566" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084566">(Apr 11 2019 at 09:33)</a>:</h4>
<p>which is kinda fun, because SyntaxTree is full-fidelity, but with token-trees we don't have whitespace, so the tree is missing all ws nodes</p>



<a name="163084597"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084597" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084597">(Apr 11 2019 at 09:33)</a>:</h4>
<p>there's another question of "parsability" of the resulting token trees</p>



<a name="163084652"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084652" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084652">(Apr 11 2019 at 09:34)</a>:</h4>
<p>ANd this is what I am not entirely sure: converting tt to <strong>Abstract</strong> syntax tree directly seems more resonable, but we don't have an AST layer between CST and HIR, and introducing one just for macros seems and overkill</p>



<a name="163084660"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084660" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084660">(Apr 11 2019 at 09:34)</a>:</h4>
<p><span class="user-mention" data-user-id="153740">@Igor Matuszewski</span> what question exactly?</p>



<a name="163084671"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084671" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084671">(Apr 11 2019 at 09:34)</a>:</h4>
<p>there are some cases in rustc macros IIRC where we produce a <code>$crate</code> quasi-keyword  which is special-cased in rustc but isn't directly parseable</p>



<a name="163084676"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084676" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084676">(Apr 11 2019 at 09:34)</a>:</h4>
<p>I think Vadim has worked on that to push back on it</p>



<a name="163084697"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084697" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084697">(Apr 11 2019 at 09:35)</a>:</h4>
<p>Oh, yeah... But, we totally can have <code>$crate</code> identifier in the syntax tree</p>



<a name="163084698"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084698" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084698">(Apr 11 2019 at 09:35)</a>:</h4>
<p>but I assume we should always have expansion that, when "pretty-printed" should be easily parseable as regular Rust, right?</p>



<a name="163084717"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084717" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084717">(Apr 11 2019 at 09:35)</a>:</h4>
<p>Not really I fear.</p>



<a name="163084720"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084720" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084720">(Apr 11 2019 at 09:35)</a>:</h4>
<p>$crate is the first reason</p>



<a name="163084726"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084726" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084726">(Apr 11 2019 at 09:35)</a>:</h4>
<p>the second reason is precedence</p>



<a name="163084738"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084738" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084738">(Apr 11 2019 at 09:35)</a>:</h4>
<p>Do we need the span about the interpolated item in final token tree ?</p>



<a name="163084817"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084817" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084817">(Apr 11 2019 at 09:36)</a>:</h4>
<p><span class="user-mention" data-user-id="153740">@Igor Matuszewski</span> basically the tree for <code>$expr * 2</code> could expand to <code>1 + 1 * 2</code>, where the tree would be right, but pretty-pritnging would be wrong</p>



<a name="163084844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084844">(Apr 11 2019 at 09:37)</a>:</h4>
<p>shouldn't pretty-printing factor in the precedence and insert disambiguating braces?</p>



<a name="163084860"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084860" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084860">(Apr 11 2019 at 09:37)</a>:</h4>
<p>so <code>(1 + 1) * 2</code></p>



<a name="163084865"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084865" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084865">(Apr 11 2019 at 09:37)</a>:</h4>
<p>(hope I'm not missing something obvious here)</p>



<a name="163084869"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084869" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084869">(Apr 11 2019 at 09:37)</a>:</h4>
<p><span class="user-mention" data-user-id="216201">@Edwin Cheng</span> yeah,  we need to correlate source spans with target spans, so that features like <code>goto definition</code> work correctly</p>



<a name="163084940"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084940" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084940">(Apr 11 2019 at 09:38)</a>:</h4>
<div class="codehilite"><pre><span></span>macro_rules! s {
    ($i:ident) =&gt; { struct $i; }
}

s!(Foo);

fn foo(_: Foo) {

}
</pre></div>



<a name="163084943"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084943" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084943">(Apr 11 2019 at 09:38)</a>:</h4>
<p>You can try this code in rust-analyzer today</p>



<a name="163084951"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084951" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084951">(Apr 11 2019 at 09:38)</a>:</h4>
<p>goto definition for <code>_: Foo</code> works</p>



<a name="163084969"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084969" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084969">(Apr 11 2019 at 09:39)</a>:</h4>
<p>And it even goes to the offset of <code>Foo</code> in <code>struct Foo</code></p>



<a name="163084973"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084973" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084973">(Apr 11 2019 at 09:39)</a>:</h4>
<p><strong>But</strong>, it does so in the wrong file :D</p>



<a name="163084982"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163084982" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163084982">(Apr 11 2019 at 09:39)</a>:</h4>
<p>So, you end up on <code>_r</code> in <code>macro_rules</code></p>



<a name="163085044"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085044" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085044">(Apr 11 2019 at 09:40)</a>:</h4>
<p>Coming back to the "easily parseable" bit, that'd allow us to bypass the "AST" if we can always produce a valid CST</p>



<a name="163085045"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085045" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085045">(Apr 11 2019 at 09:40)</a>:</h4>
<p>threading this span-remapping logic all the way to the IDE is actually one task that I feel we should do right now</p>



<a name="163085052"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085052" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085052">(Apr 11 2019 at 09:40)</a>:</h4>
<p>which also would be good, in case you'd want to syntax highlight the macro-expanded code</p>



<a name="163085053"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085053" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085053">(Apr 11 2019 at 09:40)</a>:</h4>
<p><span class="user-mention" data-user-id="153740">@Igor Matuszewski</span> yeah, I think using CST would work</p>



<a name="163085057"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085057" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085057">(Apr 11 2019 at 09:41)</a>:</h4>
<p>It's jsut that it contains "unnceseccary" bits</p>



<a name="163085075"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085075" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085075">(Apr 11 2019 at 09:41)</a>:</h4>
<p>but yeah, I also think it'll be convenient for ide features like highlighting</p>



<a name="163085098"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085098" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085098">(Apr 11 2019 at 09:42)</a>:</h4>
<p>I guess this is a good overview of what we got right now, so perhaps we shoudl proced to discussing action items?</p>



<a name="163085135"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085135" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085135">(Apr 11 2019 at 09:42)</a>:</h4>
<p>sure!</p>



<a name="163085147"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085147" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085147">(Apr 11 2019 at 09:42)</a>:</h4>
<p>sure!</p>



<a name="163085150"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085150" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085150">(Apr 11 2019 at 09:42)</a>:</h4>
<p><span class="user-mention" data-user-id="216201">@Edwin Cheng</span> do you have any questions/comments? I think that you are now more familiar with exisitng  infram than me :)</p>



<a name="163085171"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085171" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085171">(Apr 11 2019 at 09:43)</a>:</h4>
<p>For the design part i don't have any comments</p>



<a name="163085177"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085177" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085177">(Apr 11 2019 at 09:43)</a>:</h4>
<p>Good!</p>



<a name="163085185"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085185" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085185">(Apr 11 2019 at 09:43)</a>:</h4>
<p>So, I think there are two areas:</p>



<a name="163085187"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085187" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085187">(Apr 11 2019 at 09:43)</a>:</h4>
<p>But i face some problems, should i talk right now, or later?</p>



<a name="163085196"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085196" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085196">(Apr 11 2019 at 09:44)</a>:</h4>
<p><span class="user-mention" data-user-id="216201">@Edwin Cheng</span> right now is a good point probably</p>



<a name="163085255"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085255" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085255">(Apr 11 2019 at 09:44)</a>:</h4>
<p>We discussed the single char punct problem before.</p>



<a name="163085300"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085300" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085300">(Apr 11 2019 at 09:45)</a>:</h4>
<p>Currently our parser expect composed token, e.g "DOTDOTDOT" ,,</p>



<a name="163085304"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085304" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085304">(Apr 11 2019 at 09:45)</a>:</h4>
<p>right! (<span class="user-mention" data-user-id="153740">@Igor Matuszewski</span> in the TokenTree, we represent <code>&lt;&lt;</code> as two <code>&lt;</code> tokens which are <code>Joint</code>, like syn, and that is slightly different from what we do in the parser)</p>



<a name="163085374"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085374" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085374">(Apr 11 2019 at 09:46)</a>:</h4>
<p>But it make all stuffs became very complicated.</p>



<a name="163085386"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085386" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085386">(Apr 11 2019 at 09:46)</a>:</h4>
<p>Hm, perhaps we need to change the <code>TokenSource</code> interface to say</p>
<div class="codehilite"><pre><span></span>fn is_at_composed_token(tt: TokenKind)
</pre></div>


<p>?</p>



<a name="163085397"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085397" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085397">(Apr 11 2019 at 09:46)</a>:</h4>
<p>That way, we don't have to write <code>current2</code> or <code>current3</code> in the parser</p>



<a name="163085415"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085415" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085415">(Apr 11 2019 at 09:47)</a>:</h4>
<p>instead, the token source would take care of this</p>



<a name="163085439"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085439" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085439">(Apr 11 2019 at 09:47)</a>:</h4>
<p>Or should we change the parser such that it only expect what single char punct?</p>



<a name="163085457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085457">(Apr 11 2019 at 09:47)</a>:</h4>
<p>I guess <code>Is_composed_token</code> is a better apporahc</p>



<a name="163085512"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085512" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085512">(Apr 11 2019 at 09:48)</a>:</h4>
<p>than, we can change <code>TokenSource</code> to be an Iterator-like thing, instead of a slice-like thing</p>



<a name="163085532"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085532" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085532">(Apr 11 2019 at 09:48)</a>:</h4>
<p>and that should remove the need tor token-tree flattening</p>



<a name="163085539"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085539" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085539">(Apr 11 2019 at 09:48)</a>:</h4>
<p>Will it affect proc-macro?</p>



<a name="163085553"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085553" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085553">(Apr 11 2019 at 09:49)</a>:</h4>
<p>I don't think so?</p>



<a name="163085578"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085578" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085578">(Apr 11 2019 at 09:49)</a>:</h4>
<p>Another question, why we do not have a tokenstream ?</p>



<a name="163085670"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085670" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085670">(Apr 11 2019 at 09:50)</a>:</h4>
<p>tokenstream would be the "iterator-like" approach, right?</p>



<a name="163085677"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085677" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085677">(Apr 11 2019 at 09:50)</a>:</h4>
<p>THis is actually a good action item:</p>
<p>Trying something like this:</p>
<div class="codehilite"><pre><span></span>trait TOkenSOuece {
   fn current(&amp;self) -&gt; SyntaxKind;
   fn is_at_composite(&amp;self, kind: SyntaxKind);
   fn bump(&amp;mut self);
   fn bump_composite(&amp;mut sekf, SyntaxKind);
}
</pre></div>



<a name="163085775"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085775" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085775">(Apr 11 2019 at 09:52)</a>:</h4>
<p>SO, proc-macro has both a token tree and a token stream, where "stream" is the tree, but without delimiters. <span class="user-mention silent" data-user-id="123856">Vadim Petrochenkov</span>  in some issue written that this is not orthogonal, and that ideally only token tree should be enough</p>



<a name="163085844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085844">(Apr 11 2019 at 09:53)</a>:</h4>
<p>argh, can't find the comment right now</p>



<a name="163085912"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085912" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085912">(Apr 11 2019 at 09:54)</a>:</h4>
<p>but the bottom like is that</p>
<ul>
<li>we should probably switch parser to the iterator-like interface for tokens</li>
<li>we should try to avoid introducing TokenStream to the <code>ra_tt</code></li>
</ul>



<a name="163085923"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085923" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085923">(Apr 11 2019 at 09:54)</a>:</h4>
<p>I like the new trait, but how it solve the flatten problem of TokenSource &lt;-&gt; TokenTree ?</p>



<a name="163085979"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163085979" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163085979">(Apr 11 2019 at 09:55)</a>:</h4>
<p><span class="user-mention" data-user-id="216201">@Edwin Cheng</span> currently TokenSource wraps an <code>[TtToken]</code>, because we need to handle arbitrary <code>pos</code>. In the new interface, you always ask questions about the current pos, so you could use the cursor directly</p>



<a name="163086045"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086045" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086045">(Apr 11 2019 at 09:56)</a>:</h4>
<p>(<span class="user-mention" data-user-id="153740">@Igor Matuszewski</span> , if you are lost, we are talking about this interface to the parser <a href="https://github.com/rust-analyzer/rust-analyzer/blob/e6e2571bdf780d304c792d4317bbaf1d6f5d7a0a/crates/ra_parser/src/lib.rs#L32-L39" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/blob/e6e2571bdf780d304c792d4317bbaf1d6f5d7a0a/crates/ra_parser/src/lib.rs#L32-L39">https://github.com/rust-analyzer/rust-analyzer/blob/e6e2571bdf780d304c792d4317bbaf1d6f5d7a0a/crates/ra_parser/src/lib.rs#L32-L39</a>, which is random-access, while it should be iterator-ish)</p>



<a name="163086071"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086071" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086071">(Apr 11 2019 at 09:57)</a>:</h4>
<p>But do the token source only needs the top level TtToken of a TokenTree ?</p>



<a name="163086139"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086139" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086139">(Apr 11 2019 at 09:58)</a>:</h4>
<p><span class="user-mention" data-user-id="216201">@Edwin Cheng</span> sort-of.  when <code>bump</code> method would go over <code>(</code>, the token source will dive one lever deeper, by pushing a parent subtree tot he stack</p>



<a name="163086149"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086149" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086149">(Apr 11 2019 at 09:58)</a>:</h4>
<p>when we bump over <code>)</code>, the subtree is popped, and we are back to traversing the original TokenTree</p>



<a name="163086176"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086176" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086176">(Apr 11 2019 at 09:59)</a>:</h4>
<p>Ok, so we are almost out of time, let's try to summarize action-items we have</p>



<a name="163086177"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086177" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086177">(Apr 11 2019 at 09:59)</a>:</h4>
<p>So how about token-source look ahead ?</p>



<a name="163086247"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086247" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086247">(Apr 11 2019 at 10:00)</a>:</h4>
<p>I mean your parser maybe needs to look ahead some tokens ? So would we need to go backward of the cursor?</p>



<a name="163086249"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086249" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086249">(Apr 11 2019 at 10:00)</a>:</h4>
<p><span class="user-mention" data-user-id="216201">@Edwin Cheng</span> we need a constant amount of lookaheard and we don't need to look inside <code>(</code>, so that should be doable with iterato interface</p>
<ul>
<li>threading span-infromation such that goto definition example from above works.</li>
</ul>



<a name="163086277"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086277" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086277">(Apr 11 2019 at 10:01)</a>:</h4>
<p><span class="user-mention" data-user-id="153740">@Igor Matuszewski</span> would you be interested in tacking spans? That should make you touch almost every part of the analyser :D</p>



<a name="163086303"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086303" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086303">(Apr 11 2019 at 10:01)</a>:</h4>
<p>Yeah, sounds good!</p>



<a name="163086310"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086310" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086310">(Apr 11 2019 at 10:01)</a>:</h4>
<p>it'd be good to get into the nitty-gritty now ;)</p>



<a name="163086360"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086360" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086360">(Apr 11 2019 at 10:02)</a>:</h4>
<ul>
<li>trying out the new Iterator-based interface for token-source (I think <span class="user-mention" data-user-id="216201">@Edwin Cheng</span> you are basically deep in this already :) )</li>
</ul>



<a name="163086375"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086375" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086375">(Apr 11 2019 at 10:02)</a>:</h4>
<ul>
<li>actually making the mbe code handle more cases</li>
</ul>



<a name="163086386"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086386" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086386">(Apr 11 2019 at 10:02)</a>:</h4>
<p>note that 1 and 2 are sort-of ground-work for 3</p>



<a name="163086414"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086414" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086414">(Apr 11 2019 at 10:03)</a>:</h4>
<p>I actually expect that someone just throws away the expander I hacked together, and replaces it with something proper :)</p>



<a name="163086431"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086431" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086431">(Apr 11 2019 at 10:03)</a>:</h4>
<p>And <strong>that</strong> would require reading rustc very closely</p>



<a name="163086512"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086512" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086512">(Apr 11 2019 at 10:04)</a>:</h4>
<p>Should we wrap up the meeting then? (I do have some more time, so we can discuss some specific questions as well)</p>



<a name="163086513"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086513" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086513">(Apr 11 2019 at 10:04)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> yesterday i just read a lot of rustc mbe code...:(</p>



<a name="163086518"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086518" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086518">(Apr 11 2019 at 10:04)</a>:</h4>
<p>I understand that <code>:(</code> smiley :D</p>



<a name="163086622"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086622" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086622">(Apr 11 2019 at 10:06)</a>:</h4>
<p>I do have some more time too, but its up to you</p>



<a name="163086634"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086634" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086634">(Apr 11 2019 at 10:06)</a>:</h4>
<p>I'll also write meeting notes for this meetings</p>



<a name="163086652"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086652" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086652">(Apr 11 2019 at 10:07)</a>:</h4>
<p>I think it'd be good to wrap up now</p>



<a name="163086666"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086666" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086666">(Apr 11 2019 at 10:07)</a>:</h4>
<p>and come back and sync later as we go, if that makes sense</p>



<a name="163086680"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086680" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086680">(Apr 11 2019 at 10:07)</a>:</h4>
<p>yeah. Let me preemptifly create streams for span-mapping and iterator token source</p>



<a name="163086681"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086681" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086681">(Apr 11 2019 at 10:07)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> where will be the meeting notes?</p>



<a name="163086736"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086736" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086736">(Apr 11 2019 at 10:08)</a>:</h4>
<p>in the compiler-team repo I guess?</p>



<a name="163086998"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163086998" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Igor Matuszewski <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163086998">(Apr 11 2019 at 10:13)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> okay! let me know if you need some help with that</p>



<a name="163088663"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163088663" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163088663">(Apr 11 2019 at 10:41)</a>:</h4>
<p>also, random cc <span class="user-mention" data-user-id="124069">@Alexander Regueiro</span>: I saw that wg-macros thing, so it might be interested to you to see what happens with macros in RLS-2.0. Sorry for not thinking about cc-ing <em>before</em> the meeting :)</p>



<a name="163088858"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163088858" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163088858">(Apr 11 2019 at 10:45)</a>:</h4>
<p><span class="user-mention" data-user-id="153740">@Igor Matuszewski</span> here's that time-traveling problem thread: <a href="https://github.com/rust-lang/rust/pull/53778" target="_blank" title="https://github.com/rust-lang/rust/pull/53778">https://github.com/rust-lang/rust/pull/53778</a></p>



<a name="163111299"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163111299" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alexander Regueiro <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163111299">(Apr 11 2019 at 15:41)</a>:</h4>
<p>No worries. I think the plan is to keep developing macros in the current compiler alongside this effort (especially eager expansion), but it would be good to stay abreast of that. What's the timeline like, do you have an idea? <span class="user-mention" data-user-id="133169">@matklad</span></p>



<a name="163111742"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163111742" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163111742">(Apr 11 2019 at 15:45)</a>:</h4>
<p>I hope that we'll have more-or-less working subset of mbe withing months, but any further plans are unclear</p>



<a name="163794633"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163794633" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163794633">(Apr 20 2019 at 08:55)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span>  Do you have any idea how we handle $crate token in mbe expansion ?</p>



<a name="163804503"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163804503" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163804503">(Apr 20 2019 at 13:57)</a>:</h4>
<p>I would think that just creating a <code>$crate</code> identifier would work?</p>



<a name="163804554"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163804554" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163804554">(Apr 20 2019 at 13:58)</a>:</h4>
<p>so, when we expand code, and see <code>$</code> <code>crate</code> token trees, we produce a single <code>ident</code> token with <code>$crate</code> as text</p>



<a name="163807342"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163807342" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163807342">(Apr 20 2019 at 15:19)</a>:</h4>
<p>Sure, thats what i think too.  OTOH, i did manage to fix the stackoverflow bug, but i found that I don’t want to enable the <code>tt</code> matcher, it is because:</p>
<p>1. There are still some bugs in mbe which fail to expand. I will try to fix first.<br>
2. The real problem is, i found that it make the whole startup time very slow. (At least in my ancient pc), i think it related to winapi, which brings tons of structs in (previously it is def behind macro)</p>



<a name="163807425"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163807425" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163807425">(Apr 20 2019 at 15:21)</a>:</h4>
<p>yeah, I expect properly supporting macros will make perf significantly worse....</p>
<p>It might be a good idea to profile this stuff to check if its macro-expansion whihc is slow, or if it is the name resolution (which is now exercised much harder). <a href="https://github.com/AtheMathmo/cpuprofiler" target="_blank" title="https://github.com/AtheMathmo/cpuprofiler">https://github.com/AtheMathmo/cpuprofiler</a> is a useful tool for profiling</p>



<a name="163809098"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163809098" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163809098">(Apr 20 2019 at 16:09)</a>:</h4>
<blockquote>
<p><a href="https://github.com/AtheMathmo/cpuprofiler" target="_blank" title="https://github.com/AtheMathmo/cpuprofiler">https://github.com/AtheMathmo/cpuprofiler</a> is a useful tool for profiling</p>
</blockquote>
<p>It does not support Windows <span aria-label="joy" class="emoji emoji-1f602" role="img" title="joy">:joy:</span> <br>
Anyway, i would find some tools to profile it first.<br>
But my first priority is making no macro expanding errors in name resolution first. Personally i prefer a bunch of small PRs and will submit it one by one.</p>



<a name="163865599"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163865599" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> L.apz <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163865599">(Apr 21 2019 at 20:10)</a>:</h4>
<p>what kinda of test is a smoke test ? and where would it belong</p>



<a name="163867653"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163867653" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163867653">(Apr 21 2019 at 21:10)</a>:</h4>
<p><span class="user-mention" data-user-id="121360">@L.apz</span>  smoke test is the most simple test that executes given functionlity</p>



<a name="163867657"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163867657" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163867657">(Apr 21 2019 at 21:10)</a>:</h4>
<p>for macros, I'd add it to type-inference tests</p>



<a name="163867671"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163867671" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163867671">(Apr 21 2019 at 21:11)</a>:</h4>
<p>something like this</p>
<div class="codehilite"><pre><span></span>macro_rules! expr {
    ($e:expr) =&gt; { $e }
}

struct S;
fn foo() {
    let x = expr!(S);
}
</pre></div>


<p>and check that type of <code>x</code> is <code>S</code></p>



<a name="163867782"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163867782" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163867782">(Apr 21 2019 at 21:14)</a>:</h4>
<p>ah, the issue with panic b/c of local syntax ptr is fun...</p>



<a name="163867855"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163867855" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163867855">(Apr 21 2019 at 21:16)</a>:</h4>
<p>where exactly do you get the panic? I wonder if we can fix it without refactoring all the tests</p>



<a name="163869461"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163869461" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> L.apz <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163869461">(Apr 21 2019 at 21:59)</a>:</h4>
<p>I get the error when I tried to add a test within the type inference tests. The painc is caused when we iterate the types that type inference returns specifically this line : <a href="https://github.com/rust-analyzer/rust-analyzer/blob/bbc5c1d24e1a641b134f634516828301e8cfc320/crates/ra_hir/src/ty/tests.rs#L2538" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/blob/bbc5c1d24e1a641b134f634516828301e8cfc320/crates/ra_hir/src/ty/tests.rs#L2538">https://github.com/rust-analyzer/rust-analyzer/blob/bbc5c1d24e1a641b134f634516828301e8cfc320/crates/ra_hir/src/ty/tests.rs#L2538</a></p>



<a name="163869651"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163869651" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163869651">(Apr 21 2019 at 22:04)</a>:</h4>
<p>Hm, so, one fix here is to remove macro-generated exprs from SourceMap....</p>
<p>Not sure what's the best way to do that</p>



<a name="163869655"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163869655" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163869655">(Apr 21 2019 at 22:04)</a>:</h4>
<p>and also not sure what's the proper way to handle this long-term :)</p>



<a name="163869684"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163869684" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163869684">(Apr 21 2019 at 22:05)</a>:</h4>
<p>So, and absolute quick fix would be to add a <code>bool</code> flag <code>is_in_macro</code> to Expr collector, which causes us to skip source-map if it is enabled</p>



<a name="163869686"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163869686" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163869686">(Apr 21 2019 at 22:05)</a>:</h4>
<p>long-term, I guess we need to replace local ptrs in source map with global ones?</p>



<a name="163870178"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163870178" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> L.apz <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163870178">(Apr 21 2019 at 22:20)</a>:</h4>
<p>Thanks for the pointers. I'll look at it tomorrow.</p>



<a name="163895726"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163895726" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163895726">(Apr 22 2019 at 10:22)</a>:</h4>
<p><span class="user-mention" data-user-id="121360">@L.apz</span> For issue 1187, i wrote an unit test :</p>
<div class="codehilite"><pre><span></span><span class="cp">#[test]</span><span class="w"></span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">test_vec</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="n">rules</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">create_rules</span><span class="p">(</span><span class="w"></span>
<span class="w">            </span><span class="n">r</span><span class="err">#</span><span class="s">&quot;</span>
<span class="s">         macro_rules! vec {</span>
<span class="s">            ($($item:expr),*) =&gt; {</span>
<span class="s">                {</span>
<span class="s">                    let mut v = Vec::new();</span>
<span class="s">                    $(</span>
<span class="s">                        v.push($item);</span>
<span class="s">                    )*</span>
<span class="s">                    v</span>
<span class="s">                }</span>
<span class="s">            };</span>
<span class="s">}</span>
<span class="s">&quot;</span><span class="err">#</span><span class="p">,</span><span class="w"></span>
<span class="w">        </span><span class="p">);</span><span class="w"></span>
<span class="w">        </span><span class="n">assert_expansion</span><span class="p">(</span><span class="o">&amp;</span><span class="n">rules</span><span class="p">,</span><span class="w"> </span><span class="s">r#&quot;vec!();&quot;#</span><span class="p">,</span><span class="w"> </span><span class="s">r#&quot;{let mut v = Vec :: new () ;  v}&quot;#</span><span class="p">);</span><span class="w"></span>
<span class="w">        </span><span class="n">assert_expansion</span><span class="p">(</span><span class="w"></span>
<span class="w">            </span><span class="o">&amp;</span><span class="n">rules</span><span class="p">,</span><span class="w"></span>
<span class="w">            </span><span class="s">r#&quot;vec![1u32,2]&quot;#</span><span class="p">,</span><span class="w"></span>
<span class="w">            </span><span class="s">r#&quot;{let mut v = Vec :: new () ; v . push (1u32) ; v . push (2) ; v}&quot;#</span><span class="p">,</span><span class="w"></span>
<span class="w">        </span><span class="p">);</span><span class="w"></span>

<span class="w">        </span><span class="n">assert_eq</span><span class="o">!</span><span class="p">(</span><span class="w"></span>
<span class="w">            </span><span class="n">expand_to_stmts</span><span class="p">(</span><span class="o">&amp;</span><span class="n">rules</span><span class="p">,</span><span class="w"> </span><span class="s">r#&quot;vec![1u32,2]&quot;#</span><span class="p">).</span><span class="n">syntax</span><span class="p">().</span><span class="n">debug_dump</span><span class="p">().</span><span class="n">trim</span><span class="p">(),</span><span class="w"></span>
<span class="w">            </span><span class="n">r</span><span class="err">#</span><span class="s">&quot;MACRO_STMTS@[0; 45)</span>
<span class="s">  EXPR_STMT@[0; 45)</span>
<span class="s">    BLOCK_EXPR@[0; 45)</span>
<span class="s">      BLOCK@[0; 45)</span>
<span class="s">        L_CURLY@[0; 1) &quot;</span><span class="p">{</span><span class="s">&quot;</span>
<span class="s">        LET_STMT@[1; 20)</span>
<span class="s">          LET_KW@[1; 4) &quot;</span><span class="kd">let</span><span class="s">&quot;</span>
<span class="s">          BIND_PAT@[4; 8)</span>
<span class="s">            MUT_KW@[4; 7) &quot;</span><span class="k">mut</span><span class="s">&quot;</span>
<span class="s">            NAME@[7; 8)</span>
<span class="s">              IDENT@[7; 8) &quot;</span><span class="n">v</span><span class="s">&quot;</span>
<span class="s">          EQ@[8; 9) &quot;</span><span class="o">=</span><span class="s">&quot;</span>
<span class="s">          CALL_EXPR@[9; 19)</span>
<span class="s">            PATH_EXPR@[9; 17)</span>
<span class="s">              PATH@[9; 17)</span>
<span class="s">                PATH@[9; 12)</span>
<span class="s">                  PATH_SEGMENT@[9; 12)</span>
<span class="s">                    NAME_REF@[9; 12)</span>
<span class="s">                      IDENT@[9; 12) &quot;</span><span class="nb">Vec</span><span class="s">&quot;</span>
<span class="s">                COLONCOLON@[12; 14) &quot;</span>::<span class="s">&quot;</span>
<span class="s">                PATH_SEGMENT@[14; 17)</span>
<span class="s">                  NAME_REF@[14; 17)</span>
<span class="s">                    IDENT@[14; 17) &quot;</span><span class="n">new</span><span class="s">&quot;</span>
<span class="s">            ARG_LIST@[17; 19)</span>
<span class="s">              L_PAREN@[17; 18) &quot;</span><span class="p">(</span><span class="s">&quot;</span>
<span class="s">              R_PAREN@[18; 19) &quot;</span><span class="p">)</span><span class="s">&quot;</span>
<span class="s">          SEMI@[19; 20) &quot;</span><span class="p">;</span><span class="s">&quot;</span>
<span class="s">        EXPR_STMT@[20; 33)</span>
<span class="s">          METHOD_CALL_EXPR@[20; 32)</span>
<span class="s">            PATH_EXPR@[20; 21)</span>
<span class="s">              PATH@[20; 21)</span>
<span class="s">                PATH_SEGMENT@[20; 21)</span>
<span class="s">                  NAME_REF@[20; 21)</span>
<span class="s">                    IDENT@[20; 21) &quot;</span><span class="n">v</span><span class="s">&quot;</span>
<span class="s">            DOT@[21; 22) &quot;</span><span class="p">.</span><span class="s">&quot;</span>
<span class="s">            NAME_REF@[22; 26)</span>
<span class="s">              IDENT@[22; 26) &quot;</span><span class="n">push</span><span class="s">&quot;</span>
<span class="s">            ARG_LIST@[26; 32)</span>
<span class="s">              L_PAREN@[26; 27) &quot;</span><span class="p">(</span><span class="s">&quot;</span>
<span class="s">              LITERAL@[27; 31)</span>
<span class="s">                INT_NUMBER@[27; 31) &quot;</span><span class="mi">1</span><span class="k">u32</span><span class="s">&quot;</span>
<span class="s">              R_PAREN@[31; 32) &quot;</span><span class="p">)</span><span class="s">&quot;</span>
<span class="s">          SEMI@[32; 33) &quot;</span><span class="p">;</span><span class="s">&quot;</span>
<span class="s">        EXPR_STMT@[33; 43)</span>
<span class="s">          METHOD_CALL_EXPR@[33; 42)</span>
<span class="s">            PATH_EXPR@[33; 34)</span>
<span class="s">              PATH@[33; 34)</span>
<span class="s">                PATH_SEGMENT@[33; 34)</span>
<span class="s">                  NAME_REF@[33; 34)</span>
<span class="s">                    IDENT@[33; 34) &quot;</span><span class="n">v</span><span class="s">&quot;</span>
<span class="s">            DOT@[34; 35) &quot;</span><span class="p">.</span><span class="s">&quot;</span>
<span class="s">            NAME_REF@[35; 39)</span>
<span class="s">              IDENT@[35; 39) &quot;</span><span class="n">push</span><span class="s">&quot;</span>
<span class="s">            ARG_LIST@[39; 42)</span>
<span class="s">              L_PAREN@[39; 40) &quot;</span><span class="p">(</span><span class="s">&quot;</span>
<span class="s">              LITERAL@[40; 41)</span>
<span class="s">                INT_NUMBER@[40; 41) &quot;</span><span class="mi">2</span><span class="s">&quot;</span>
<span class="s">              R_PAREN@[41; 42) &quot;</span><span class="p">)</span><span class="s">&quot;</span>
<span class="s">          SEMI@[42; 43) &quot;</span><span class="p">;</span><span class="s">&quot;</span>
<span class="s">        PATH_EXPR@[43; 44)</span>
<span class="s">          PATH@[43; 44)</span>
<span class="s">            PATH_SEGMENT@[43; 44)</span>
<span class="s">              NAME_REF@[43; 44)</span>
<span class="s">                IDENT@[43; 44) &quot;</span><span class="n">v</span><span class="s">&quot;</span>
<span class="s">        R_CURLY@[44; 45) &quot;</span><span class="p">}</span><span class="s">&quot;&quot;</span><span class="err">#</span><span class="w"></span>
<span class="w">        </span><span class="p">);</span><span class="w"></span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>
</pre></div>



<a name="163895820"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163895820" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163895820">(Apr 22 2019 at 10:25)</a>:</h4>
<p>And its pass, so..... I would want to know how do  you expand your macro ? Maybe related to be used an incorrect <code>token_tree_to_xx</code> ?</p>



<a name="163897242"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163897242" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> L.apz <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163897242">(Apr 22 2019 at 11:06)</a>:</h4>
<p><span class="user-mention" data-user-id="216201">@Edwin Cheng</span>   The method is used was <code>mbe::token_tree_to_expr</code>.The error  also only shows up when you debug the extension and check the debug output  otherwise it works fine.</p>



<a name="163899875"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/163899875" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#163899875">(Apr 22 2019 at 12:13)</a>:</h4>
<p>I just tried <code>mbe::token_tree_to_expr</code> one, and it works too. Seem like if it is a parsing error, it should be appeared in other places. Let see if it appear again and i will fix it. :)</p>



<a name="164001487"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164001487" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164001487">(Apr 23 2019 at 15:43)</a>:</h4>
<p>I have a stupid question:</p>
<p>in <a href="https://github.com/rust-lang/rust/blob/5d20ff4d2718c820632b38c1e49d4de648a9810b/src/libcore/internal_macros.rs#L81" target="_blank" title="https://github.com/rust-lang/rust/blob/5d20ff4d2718c820632b38c1e49d4de648a9810b/src/libcore/internal_macros.rs#L81">https://github.com/rust-lang/rust/blob/5d20ff4d2718c820632b38c1e49d4de648a9810b/src/libcore/internal_macros.rs#L81</a><br>
It defined a mbe :</p>
<div class="codehilite"><pre><span></span><span class="n">macro_rules</span><span class="o">!</span><span class="w"> </span><span class="n">impl_fn_for_zst</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="p">(</span><span class="cp">$(</span><span class="w"></span>
<span class="w">        </span><span class="cp">$(</span><span class="w"> </span><span class="cp">#[$attr: meta]</span><span class="w"> </span><span class="p">)</span><span class="o">*</span><span class="w"></span>
<span class="w">        </span><span class="k">struct</span> <span class="cp">$Name</span>: <span class="nc">ident</span><span class="w"> </span><span class="k">impl</span><span class="cp">$(</span><span class="w"> </span><span class="o">&lt;</span><span class="cp">$(</span><span class="w"> </span><span class="cp">$lifetime</span><span class="w"> </span>: <span class="nc">lifetime</span><span class="w"> </span><span class="p">),</span><span class="o">+&gt;</span><span class="w"> </span><span class="p">)</span><span class="o">?</span><span class="w"> </span><span class="nb">Fn</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
<span class="w">            </span><span class="o">|</span><span class="cp">$(</span><span class="w"> </span><span class="cp">$arg</span>: <span class="nc">ident</span>: <span class="cp">$ArgTy</span>: <span class="nc">ty</span><span class="w"> </span><span class="p">),</span><span class="o">*|</span><span class="w"> </span>-&gt; <span class="cp">$ReturnTy</span>: <span class="nc">ty</span><span class="w"></span>
<span class="w">            </span><span class="cp">$body</span>: <span class="nc">block</span><span class="p">;</span><span class="w"></span>
<span class="w">    </span><span class="p">)</span><span class="o">+</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">       </span><span class="p">....</span><span class="w"></span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>


<p>and in <a href="https://github.com/rust-lang/rust/blob/316a391dcb7d66dc25f1f9a4ec9d368ef7615005/src/libcore/str/mod.rs#L4114" target="_blank" title="https://github.com/rust-lang/rust/blob/316a391dcb7d66dc25f1f9a4ec9d368ef7615005/src/libcore/str/mod.rs#L4114">https://github.com/rust-lang/rust/blob/316a391dcb7d66dc25f1f9a4ec9d368ef7615005/src/libcore/str/mod.rs#L4114</a><br>
It invoke this macro by:</p>
<div class="codehilite"><pre><span></span><span class="n">impl_fn_for_zst</span><span class="o">!</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="cp">#[derive(Clone)]</span><span class="w"></span>
<span class="w">    </span><span class="k">struct</span> <span class="nc">CharEscapeDebugContinue</span><span class="w"> </span><span class="k">impl</span><span class="w"> </span><span class="nb">Fn</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">|</span><span class="n">c</span>: <span class="nc">char</span><span class="o">|</span><span class="w"> </span>-&gt; <span class="nc">char</span>::<span class="n">EscapeDebug</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="n">c</span><span class="p">.</span><span class="n">escape_debug_ext</span><span class="p">(</span><span class="kc">false</span><span class="p">)</span><span class="w"></span>
<span class="w">    </span><span class="p">};</span><span class="w"></span>

<span class="w">    </span><span class="cp">#[derive(Clone)]</span><span class="w"></span>
<span class="w">    </span><span class="k">struct</span> <span class="nc">CharEscapeUnicode</span><span class="w"> </span><span class="k">impl</span><span class="w"> </span><span class="nb">Fn</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">|</span><span class="n">c</span>: <span class="nc">char</span><span class="o">|</span><span class="w"> </span>-&gt; <span class="nc">char</span>::<span class="n">EscapeUnicode</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="n">c</span><span class="p">.</span><span class="n">escape_unicode</span><span class="p">()</span><span class="w"></span>
<span class="w">    </span><span class="p">};</span><span class="w"></span>
<span class="w">    </span><span class="cp">#[derive(Clone)]</span><span class="w"></span>
<span class="w">    </span><span class="k">struct</span> <span class="nc">CharEscapeDefault</span><span class="w"> </span><span class="k">impl</span><span class="w"> </span><span class="nb">Fn</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">|</span><span class="n">c</span>: <span class="nc">char</span><span class="o">|</span><span class="w"> </span>-&gt; <span class="nc">char</span>::<span class="n">EscapeDefault</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="n">c</span><span class="p">.</span><span class="n">escape_default</span><span class="p">()</span><span class="w"></span>
<span class="w">    </span><span class="p">};</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>


<p>A part of macro rules is <code> |$( $arg: ident: $ArgTy: ty ),*| -&gt; $ReturnTy: ty</code> and it try to match <code>|c: char| -&gt; char::EscapeUnicode</code>,<br>
but noted that the comma have to be presented in the macro invocation, but it is missing here. <br>
Do i miss missing something here???</p>



<a name="164001576"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164001576" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164001576">(Apr 23 2019 at 15:44)</a>:</h4>
<p><code>*</code> is a special repetition syntax in mbe</p>



<a name="164001615"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164001615" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164001615">(Apr 23 2019 at 15:45)</a>:</h4>
<p><code>$(foo),*</code> means <em>separated</em> by <code>,</code>,  so, trailing comma is forbidden</p>



<a name="164001634"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164001634" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164001634">(Apr 23 2019 at 15:45)</a>:</h4>
<p><code>$(foo,)*</code> is <em>terminated</em> by <code>,</code>, ie, trailing comma is required</p>



<a name="164001696"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164001696" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164001696">(Apr 23 2019 at 15:46)</a>:</h4>
<p><code>$(foo),*,?</code> I think is more or less "separated by <code>,</code>, with an optional trailing comma"</p>



<a name="164001715"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164001715" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164001715">(Apr 23 2019 at 15:46)</a>:</h4>
<p>though, iiuc, the last one can parse a comma by itself</p>



<a name="164001823"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164001823" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164001823">(Apr 23 2019 at 15:47)</a>:</h4>
<p>So it is optional ?? Do you have any document/ material i can read about that?</p>



<a name="164001883"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164001883" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164001883">(Apr 23 2019 at 15:48)</a>:</h4>
<p>The <em>trailing</em> one is optional,</p>



<a name="164002042"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164002042" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164002042">(Apr 23 2019 at 15:50)</a>:</h4>
<p>I guess the docs are <a href="https://doc.rust-lang.org/reference/macros-by-example.html" target="_blank" title="https://doc.rust-lang.org/reference/macros-by-example.html">https://doc.rust-lang.org/reference/macros-by-example.html</a></p>



<a name="164002069"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164002069" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164002069">(Apr 23 2019 at 15:50)</a>:</h4>
<p><a href="https://danielkeep.github.io/tlborm/book/README.html" target="_blank" title="https://danielkeep.github.io/tlborm/book/README.html">https://danielkeep.github.io/tlborm/book/README.html</a> used to be a good resource, but it hasn't keep up with language changes</p>



<a name="164002437"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164002437" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164002437">(Apr 23 2019 at 15:55)</a>:</h4>
<p>I did read both actually, but no one mention the seperator itself in pattern is optional. :(</p>



<a name="164002634"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164002634" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164002634">(Apr 23 2019 at 15:56)</a>:</h4>
<p>there's a bit about this here: <a href="https://danielkeep.github.io/tlborm/book/pat-trailing-separators.html" target="_blank" title="https://danielkeep.github.io/tlborm/book/pat-trailing-separators.html">https://danielkeep.github.io/tlborm/book/pat-trailing-separators.html</a></p>



<a name="164002770"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164002770" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164002770">(Apr 23 2019 at 15:58)</a>:</h4>
<p>Seem to be i am missing that part. Thanks alot.</p>



<a name="164002811"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164002811" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164002811">(Apr 23 2019 at 15:58)</a>:</h4>
<p>"Note that this cannot be used in all contexts. If the compiler rejects this, you will likely need to use multiple arms and/or incremental matching" :(</p>



<a name="164002942"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164002942" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164002942">(Apr 23 2019 at 16:00)</a>:</h4>
<p>Yeah, I personally see <code>,*</code> thing as one of the (lesser) warts of the existing macro systems. Everywhere else we are careful to allow trailing separators, but in macros they are forbidden!</p>
<p>It used to be the case that most stdlib macros forbade trailing <code>,</code> :D</p>



<a name="164003015"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003015" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003015">(Apr 23 2019 at 16:00)</a>:</h4>
<p>But actually , the link you sent me is actually talking about the question i mention. Thats why in the link they needs to use <code>($($exprs:expr),* $(,)*) =&gt; {...};</code> to overcome that.</p>



<a name="164003203"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003203" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003203">(Apr 23 2019 at 16:02)</a>:</h4>
<p>So back to origin, WHY it is optional ? I still don't get it</p>



<a name="164003458"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003458" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003458">(Apr 23 2019 at 16:05)</a>:</h4>
<p>It is not really optional</p>



<a name="164003463"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003463" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003463">(Apr 23 2019 at 16:05)</a>:</h4>
<p>let me give you a couple of examples</p>



<a name="164003481"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003481" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003481">(Apr 23 2019 at 16:05)</a>:</h4>
<p>let's say we have pattern <code>$(a),*</code></p>



<a name="164003534"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003534" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003534">(Apr 23 2019 at 16:06)</a>:</h4>
<p>here are some cases where it matches succesfully</p>



<a name="164003556"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003556" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003556">(Apr 23 2019 at 16:06)</a>:</h4>
<p>1. <code> </code><br>
2. <code>a</code><br>
3. <code>a, a</code><br>
4. <code>a, a, a</code></p>



<a name="164003594"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003594" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003594">(Apr 23 2019 at 16:06)</a>:</h4>
<p>And here are some <strong>failed</strong> cases</p>



<a name="164003628"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003628" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003628">(Apr 23 2019 at 16:07)</a>:</h4>
<p>1. <code>a, </code> // tariling separator<br>
2. <code>,</code>   // only separator<br>
3. <code>a a</code> // no separator</p>



<a name="164003638"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003638" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003638">(Apr 23 2019 at 16:07)</a>:</h4>
<p>does this make sense?</p>



<a name="164003759"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003759" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003759">(Apr 23 2019 at 16:08)</a>:</h4>
<p>So what you mean is, the  kleene operator is applied on the comma, not the whole pattern, right ?</p>



<a name="164003906"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003906" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003906">(Apr 23 2019 at 16:10)</a>:</h4>
<p>no, it is applied to <em>both</em> pattern and comma</p>



<a name="164003938"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003938" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> kennytm <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003938">(Apr 23 2019 at 16:10)</a>:</h4>
<p><code>$(a),*</code> means matching <code>a $(,a)*</code> or matching empty</p>



<a name="164003945"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164003945" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164003945">(Apr 23 2019 at 16:10)</a>:</h4>
<p>the important thing is the diffrence between <code>$(a,)*</code> and <code>$(a),*</code></p>



<a name="164004272"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164004272" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164004272">(Apr 23 2019 at 16:14)</a>:</h4>
<p>Okay, i seem to get it, but why the book mentions <code>$($exprs:expr),* $(,)*) =&gt; {...};</code> ??</p>



<a name="164004276"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164004276" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> kennytm <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164004276">(Apr 23 2019 at 16:14)</a>:</h4>
<p>or, read it this way,</p>
<ul>
<li><code>$($stuff)§*</code> means matching Zero Or More <code>$stuff</code>, and between every <code>$stuff</code> there must be a <code>§</code></li>
<li><code>$($stuff)*</code> means matching Zero Or More <code>$stuff</code>, and between every <code>$stuff</code> there must be (nothing)</li>
<li><code>$($stuff)§+</code> means matching One Or More <code>$stuff</code>, and between every <code>$stuff</code> there must be a <code>§</code></li>
<li><code>$($stuff)+</code> means matching One Or More <code>$stuff</code>, and between every <code>$stuff</code> there must be (nothing)</li>
<li><code>$($stuff)?</code> means matching Zero Or One <code>$stuff</code>.</li>
</ul>



<a name="164004335"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164004335" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> kennytm <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164004335">(Apr 23 2019 at 16:15)</a>:</h4>
<p>so <code>$($exprs:expr),*</code> means matching Zero Or More expressions, with a <code>,</code> between each expression</p>



<a name="164004357"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164004357" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> kennytm <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164004357">(Apr 23 2019 at 16:15)</a>:</h4>
<p>but this can't match the trailing comma in</p>
<div class="codehilite"><pre><span></span>1,2,3,4,
//     ^
</pre></div>



<a name="164004495"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164004495" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> kennytm <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164004495">(Apr 23 2019 at 16:17)</a>:</h4>
<p>so a <code>$(,)*</code> is added after <code>$($exprs:expr),*</code> to match that trailing comma as well (should be written as <code>$(,)?</code> nowadays but it's a relatively new feature).</p>



<a name="164004518"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164004518" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164004518">(Apr 23 2019 at 16:17)</a>:</h4>
<p>Oh i see, that's because it is allowed for the trailing comma .. I just misunderstand that in the opposite way .. I thought it is because it allow no trailing comma.</p>



<a name="164004661"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164004661" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164004661">(Apr 23 2019 at 16:19)</a>:</h4>
<p>Thank you so much you guys , <span aria-label="heart" class="emoji emoji-2764" role="img" title="heart">:heart:</span></p>



<a name="164024743"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164024743" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> L.apz <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164024743">(Apr 23 2019 at 20:01)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span>  I'm looking at adding goto for macros but I keep on a getting some type inference errors</p>
<div class="codehilite"><pre><span></span>no method named `parse` found for type `&amp;db::RootDatabase` in the current scope
</pre></div>


<p>Do you know what causes this or have any idea on something that I'm missing</p>



<a name="164024829"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164024829" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164024829">(Apr 23 2019 at 20:02)</a>:</h4>
<p>sigh....</p>
<p>That's a rustc bug with incremental compilation, <code>rm target/debug/.incremental -rf</code> should fix it</p>



<a name="164024836"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164024836" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164024836">(Apr 23 2019 at 20:02)</a>:</h4>
<p>temporarly :)</p>



<a name="164024894"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164024894" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> L.apz <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164024894">(Apr 23 2019 at 20:03)</a>:</h4>
<p>thanks</p>



<a name="164027508"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164027508" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Kolb <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164027508">(Apr 23 2019 at 20:37)</a>:</h4>
<p>I usually encounter this in<code>ra_ide_api</code> so you only need to delete that one</p>



<a name="164142673"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164142673" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164142673">(Apr 25 2019 at 04:35)</a>:</h4>
<p>Just want to share with you gus, while i add tests to mbe by dogfooding all macro in <code>ra</code>, i found these two "amazing" macros:</p>
<ul>
<li><a href="https://github.com/retep998/winapi-rs/blob/a7ef2bca086aae76cf6c4ce4c2552988ed9798ad/src/um/d3d11sdklayers.rs#L163" target="_blank" title="https://github.com/retep998/winapi-rs/blob/a7ef2bca086aae76cf6c4ce4c2552988ed9798ad/src/um/d3d11sdklayers.rs#L163">https://github.com/retep998/winapi-rs/blob/a7ef2bca086aae76cf6c4ce4c2552988ed9798ad/src/um/d3d11sdklayers.rs#L163</a></li>
<li><a href="https://github.com/retep998/winapi-rs/blob/a7ef2bca086aae76cf6c4ce4c2552988ed9798ad/src/um/d3d12sdklayers.rs#L169" target="_blank" title="https://github.com/retep998/winapi-rs/blob/a7ef2bca086aae76cf6c4ce4c2552988ed9798ad/src/um/d3d12sdklayers.rs#L169">https://github.com/retep998/winapi-rs/blob/a7ef2bca086aae76cf6c4ce4c2552988ed9798ad/src/um/d3d12sdklayers.rs#L169</a></li>
</ul>
<p>I don't know how to handle these cases, should we just ignore it or still parse it ?</p>



<a name="164145851"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164145851" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164145851">(Apr 25 2019 at 05:59)</a>:</h4>
<p>Oh, hellow winapi, my old friend! (<a href="https://github.com/intellij-rust/intellij-rust/blob/53b67c89eae86969bf244683aa5b5ccb2639d6b0/src/test/kotlin/org/rustSlowTests/CargoProjectResolveTest.kt#L216" target="_blank" title="https://github.com/intellij-rust/intellij-rust/blob/53b67c89eae86969bf244683aa5b5ccb2639d6b0/src/test/kotlin/org/rustSlowTests/CargoProjectResolveTest.kt#L216">https://github.com/intellij-rust/intellij-rust/blob/53b67c89eae86969bf244683aa5b5ccb2639d6b0/src/test/kotlin/org/rustSlowTests/CargoProjectResolveTest.kt#L216</a>)</p>



<a name="164145866"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164145866" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164145866">(Apr 25 2019 at 05:59)</a>:</h4>
<p>Long term, we definitely should just expand them</p>



<a name="164145938"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164145938" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164145938">(Apr 25 2019 at 06:01)</a>:</h4>
<p>Medium term, it’s okay if we fail to expand them, as long as the analyzer doesn’t hang</p>



<a name="164190729"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164190729" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164190729">(Apr 25 2019 at 17:24)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span>  I have another mbe design problem want to discuss :</p>



<a name="164190778"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164190778" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164190778">(Apr 25 2019 at 17:24)</a>:</h4>
<p>(deleted)</p>



<a name="164191166"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164191166" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164191166">(Apr 25 2019 at 17:29)</a>:</h4>
<p>## Problems</p>
<p>Who do will handle multiple <code>punct</code> in mbe ?</p>
<p>### Lexer</p>
<p>Currently our lexer produces 2 kinds of Punct :</p>
<ul>
<li>
<p>Single character punct (type I)<br>
    * <code>;,(){}[]&lt;&gt;@#~?$&amp;|+*/^%-</code><br>
    * <code>.:=!-</code></p>
</li>
<li>
<p>Multiple characters punct (type II) <br>
    * <code>..</code> <code>...</code> <code>::</code> <code>==</code> <code>=&gt;</code> <code>!=</code> <code>-&gt;</code></p>
</li>
<li>
<p>But it <strong>DO NOT</strong> produces followings <strong>multiple characters punct</strong>: (type III)<br>
    * <code>&lt;=</code> <code>&gt;=</code> <code>+=</code> <code>-=</code> <code>|=</code> <code>&amp;=</code> <code>^=</code> <code>/=</code> <code>*=</code> <code>%=</code> <code>&amp;&amp;</code> <code>||</code> <code>&lt;&lt;</code> <code>&gt;&gt;</code> <code>&lt;&lt;=</code> <code>&gt;&gt;=</code></p>
</li>
</ul>
<p>### How current mbe macro invocation works</p>
<p>1. When parser see a macro call, it will parse all its argument by <code>ast::token_tree</code>, which basically just use what lexer produced to make a tree. All whitespace information are preserved in this phase and only <strong>type I</strong> and <strong>type II</strong> punct will show up in <code>ast::token_tree</code>.<br>
2. When <code>hir</code> expands a macro, it calls <code>mbe::ast_to_token_tree</code> to convert it to a <code>tt::TokenTree</code>. Although <code>tt::TokenTree</code> only store <strong>type I</strong> punct, while we extract <strong>type II</strong> punct to <strong>type I</strong> , we will store <code>is_joint_next</code> to preserve the whitespace information. But, this method cannot preverse white space information of <strong>type III</strong> . We have to look ahead to see if whether we need whitespace:</p>
<div class="codehilite"><pre><span></span>// Handle type II white space
for char in token.text().chars() {
    if let Some(char) = prev {
        token_trees.push(
            tt::Leaf::from(tt::Punct { char, spacing: tt::Spacing::Joint })
                .into(),
        );
    }
    prev = Some(char)
}
// Handle type III white space
if let Some(char) = prev {
    let spacing = match child_iter.peek() {
        Some(SyntaxElement::Token(token)) =&gt; {
            if token.kind().is_punct() {
                tt::Spacing::Joint
            } else {
                tt::Spacing::Alone
            }
        }
        _ =&gt; tt::Spacing::Alone,
    };

    token_trees.push(tt::Leaf::from(tt::Punct { char, spacing }).into());
}
</pre></div>


<p>3. While mbe match rule matchers, (e.g. <code>$path:path</code>), it will call <code>parse_xxx</code> from <code>ra_syntax</code> crate as a blackbox parser, So that's why the <code>type III</code> white space information above will be needed.<br>
4. After macro expansion is finished, a <code>tt::Subtree</code> will be returned. And hir will call <code>mbe::token_tree_to_xxx</code> to convert it back to <code>SyntaxNode</code> tree. And <strong>all whitespace information will be lost</strong> while conversion. There are two cases:<br>
a. <code>tt::TokenTree</code> contains no other macro calls. That's fine as we do not need these information any more. (in mbe point of view)<br>
b. <code>tt::TokenTree</code> contains other macro calls. <code>hir</code> will try to exapnd it again. (Goto step 1). However, all whitespace information are lost, so we have a trouble. </p>
<p>Imagine following case:</p>
<div class="codehilite"><pre><span></span><span class="n">macro_rules</span><span class="o">!</span><span class="w"> </span><span class="n">foo</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="p">()</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="n">bar</span><span class="o">!</span><span class="p">[</span><span class="w"></span>
<span class="w">        </span><span class="k">struct</span> <span class="nc">Foo</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">            </span><span class="n">problem</span>: ::<span class="n">other_crate</span>::<span class="k">type</span><span class="p">,</span><span class="w"></span>
<span class="w">        </span><span class="p">}]</span><span class="w"></span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>

<span class="n">macro_rules</span><span class="o">!</span><span class="w"> </span><span class="n">bar</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="p">((</span><span class="cp">$tt</span>:<span class="nc">tt</span><span class="p">)</span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="o">=&gt;</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="c1">// Something..</span>
<span class="w">        </span><span class="p">......</span><span class="w"></span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>


<p>Note that after <code>bar</code> macro expansion, the <code>SyntaxNode</code> of this line:<br>
<code>problem: ::other_crate::type</code><br>
is <code>[IDENT, COLON, COLONCOLON, ...]</code>. </p>
<p>However, when we try to expand <code>bar</code> using algorithm in step 2,  the <code>tt::TokenTree</code> will become:<br>
<code> [IDENT, COLON(joint_to_next=true),COLON(joint_to_next=true), COLON(joint_to_next=true)] </code><br>
 And it is simply wrong.</p>



<a name="164191541"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164191541" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164191541">(Apr 25 2019 at 17:33)</a>:</h4>
<p>I think completely loosing whitespace info should be ok for macro expansion</p>



<a name="164191564"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164191564" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164191564">(Apr 25 2019 at 17:33)</a>:</h4>
<p>However, the fact that the lexer produces both joint and non-joint tokens is not</p>



<a name="164191580"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164191580" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164191580">(Apr 25 2019 at 17:34)</a>:</h4>
<p>I think we should fix the lexer to always produce single-character tokesn</p>



<a name="164191873"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164191873" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164191873">(Apr 25 2019 at 17:37)</a>:</h4>
<p>But the last example still cannot resolve, right ?<br>
even the lexer is  only produce single char token, the <code>SyntaxNode </code>will be <code>[IDENT, COLON, COLON COLON]</code>...</p>



<a name="164192167"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164192167" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164192167">(Apr 25 2019 at 17:40)</a>:</h4>
<p>Hm, riiight....</p>



<a name="164192259"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164192259" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164192259">(Apr 25 2019 at 17:41)</a>:</h4>
<p>The brute-force solution would be to insert whitespace when we go from tt to ast</p>



<a name="164192321"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164192321" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164192321">(Apr 25 2019 at 17:42)</a>:</h4>
<p>Like, initially, for foo, we have syntax tree from ws typed by the user</p>



<a name="164192343"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164192343" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164192343">(Apr 25 2019 at 17:42)</a>:</h4>
<p>when we cook tt out of it, we get <code>:, join_to_next=false</code></p>



<a name="164192383"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164192383" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164192383">(Apr 25 2019 at 17:43)</a>:</h4>
<p>whn we then expand foo, we preserve this token and get join_to_next=false in the expantion</p>



<a name="164192388"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164192388" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164192388">(Apr 25 2019 at 17:43)</a>:</h4>
<p>which is ok</p>



<a name="164192441"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164192441" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164192441">(Apr 25 2019 at 17:43)</a>:</h4>
<p>what is not ok is that, when we try to parse that token tree as an item list, we render it into a tree without a space (whihc is what you are saying)</p>



<a name="164192456"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164192456" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164192456">(Apr 25 2019 at 17:43)</a>:</h4>
<p>I think we can tweak that last step to insert a dummy space in that case?</p>



<a name="164192522"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164192522" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164192522">(Apr 25 2019 at 17:44)</a>:</h4>
<p>that is, we don't need to preserve vs exactly as the user typed them</p>



<a name="164192551"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164192551" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164192551">(Apr 25 2019 at 17:44)</a>:</h4>
<p>we just need <em>some</em> spacing which maintains jointness</p>



<a name="164192658"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164192658" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164192658">(Apr 25 2019 at 17:45)</a>:</h4>
<p>Yes we can :  </p>
<div class="codehilite"><pre><span></span>impl&lt;&#39;a, Q: Querier&gt; TreeSink for TtTreeSink&lt;&#39;a, Q&gt; {
    fn token(&amp;mut self, kind: SyntaxKind, n_tokens: u8) {
        if kind == L_DOLLAR || kind == R_DOLLAR {
            self.token_pos += n_tokens as usize;
            return;
        }

        for _ in 0..n_tokens {
            self.buf += &amp;self.src_querier.token(self.token_pos).1;
            self.token_pos += 1;
        }
        self.text_pos += TextUnit::of_str(&amp;self.buf);
        let text = SmolStr::new(self.buf.as_str());
        self.buf.clear();
        self.inner.token(kind, text);

        // Add a white space to token
        let (last_kind, _, last_joint_to_next ) = self.src_querier.token(self.token_pos-n_tokens as usize);
        if !last_joint_to_next &amp;&amp; last_kind.is_punct() {
            let (cur_kind, _, _ ) = self.src_querier.token(self.token_pos);
            if cur_kind.is_punct() {
                self.inner.token(WHITESPACE, &quot; &quot;.into());
            }
        }
    }
</pre></div>



<a name="164192740"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164192740" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164192740">(Apr 25 2019 at 17:46)</a>:</h4>
<p>It is the code we transform <code>tt</code> to syntax node by <code>SyntaxNodeBuilder</code></p>



<a name="164194022"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164194022" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164194022">(Apr 25 2019 at 18:00)</a>:</h4>
<p>However, as you said, we should start from making the lexer only product single char. Let me push a PR for the bugs fixes i did in last few days first. And make the lexer PR afterward.</p>



<a name="164194213"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164194213" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164194213">(Apr 25 2019 at 18:02)</a>:</h4>
<p>I also wonder what we shoud do first:</p>
<ul>
<li>refactoring to produce decomposed tokens</li>
<li>refactoring to make <code>TokenSource</code> and iterator, instead of a random-access sequecne</li>
</ul>



<a name="164194248"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164194248" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164194248">(Apr 25 2019 at 18:02)</a>:</h4>
<p>I have a theory that the first one would be easier if we do the second one first</p>



<a name="164194792"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164194792" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164194792">(Apr 25 2019 at 18:08)</a>:</h4>
<p>The first one is quite straight forward. But the second one we still need to think about the design. The iterator will need to be able <code>peek(3)</code>, and i still need to travesal a tree by index. Hm....</p>



<a name="164194890"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164194890" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164194890">(Apr 25 2019 at 18:09)</a>:</h4>
<p>yeah, I haven't looked closely at that code for a while, so I trust your judgment here!</p>



<a name="164427237"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164427237" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164427237">(Apr 29 2019 at 05:56)</a>:</h4>
<p>Just found that it is a valid macro: </p>
<div class="codehilite"><pre><span></span>macro_rules! foo {
    ($a:ident, $b:ident, $c:tt) =&gt; {

        macro_rules! bar {
            ($bi:ident) =&gt; {
                fn $bi() -&gt; u8 {$c}
            }
        }

        bar!($a);
        fn $b() -&gt; u8 {$c}
    }
}

foo!(x,y, 1);

fn main() {
    println!(&quot;{}&quot;, x() + y());
}
</pre></div>



<a name="164434768"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164434768" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164434768">(Apr 29 2019 at 08:41)</a>:</h4>
<p>oh yeah, macros can totally declare macros</p>



<a name="164446835"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164446835" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164446835">(Apr 29 2019 at 12:22)</a>:</h4>
<p>Finally I fixed all dogfooding bugs in mbe, except <code>doc comment</code> =&gt;  $meta . <br>
Here is the log of errror : <a href="https://gist.github.com/edwin0cheng/3f3f17bba11b8c262bf375895379a60e" target="_blank" title="https://gist.github.com/edwin0cheng/3f3f17bba11b8c262bf375895379a60e">https://gist.github.com/edwin0cheng/3f3f17bba11b8c262bf375895379a60e</a></p>
<p>Just curious,   how would we handle <code>DocComment</code> ?</p>
<ul>
<li>Do lexer produce a DOC_COMMENT SyntaxKind ?</li>
<li>How kind of <code>tt::TokenTree</code> should it be ?</li>
</ul>



<a name="164450357"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164450357" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164450357">(Apr 29 2019 at 13:08)</a>:</h4>
<p>I think we should deshugar doc comments to <code>#[doc = "..."]</code> attributes when we convert syntax tree to tt</p>



<a name="164450397"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164450397" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164450397">(Apr 29 2019 at 13:08)</a>:</h4>
<p>conversely, when we parse tt as syntax tree, there would be no doc comments, only attributes</p>



<a name="164451576"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164451576" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164451576">(Apr 29 2019 at 13:24)</a>:</h4>
<p>This is a solution I have never not thought of... nice</p>



<a name="164451830"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164451830" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164451830">(Apr 29 2019 at 13:27)</a>:</h4>
<p>that is basically what rustc's parser is doig</p>



<a name="164852432"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164852432" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164852432">(May 04 2019 at 06:04)</a>:</h4>
<p>Do anyone know a crate /regex to escape a double quote string and unescape a double quote string but leave other escape char intact? <br>
e.g. <code>I said: "she said\" I said \" "\n</code> &lt;-&gt; <code>I said: \"she said\\\" I said \\\" \"\n</code></p>



<a name="164856569"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164856569" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164856569">(May 04 2019 at 08:10)</a>:</h4>
<p>Perhaps <a href="https://doc.rust-lang.org/std/primitive.str.html#method.escape_default" target="_blank" title="https://doc.rust-lang.org/std/primitive.str.html#method.escape_default">https://doc.rust-lang.org/std/primitive.str.html#method.escape_default</a> culd help?</p>



<a name="164856997"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164856997" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164856997">(May 04 2019 at 08:24)</a>:</h4>
<p>That one will escape all things, including <code>\n</code> etc.. i was thinking whether we only want to escape double quotes in doc comment de sugaring.</p>



<a name="164857114"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164857114" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164857114">(May 04 2019 at 08:28)</a>:</h4>
<p>Hm, I think we need to escape all things?</p>



<a name="164857117"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164857117" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164857117">(May 04 2019 at 08:28)</a>:</h4>
<p>Like, doc comments are not guranteeed to be valid literals</p>



<a name="164857129"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164857129" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164857129">(May 04 2019 at 08:29)</a>:</h4>
<div class="codehilite"><pre><span></span>/// invalid escape: \u{zzzzz}
fn foo()
</pre></div>



<a name="164857337"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164857337" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164857337">(May 04 2019 at 08:35)</a>:</h4>
<p>Good point！</p>



<a name="164927762"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164927762" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164927762">(May 05 2019 at 17:09)</a>:</h4>
<p>I am hacking on salsa now a bit, and I am pleasantly surprised by how far our support for macros has gone!</p>



<a name="164927763"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164927763" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164927763">(May 05 2019 at 17:09)</a>:</h4>
<p><a href="https://youtu.be/akdZefuvBAQ" target="_blank" title="https://youtu.be/akdZefuvBAQ">https://youtu.be/akdZefuvBAQ</a></p>
<div class="youtube-video message_inline_image"><a data-id="akdZefuvBAQ" href="https://youtu.be/akdZefuvBAQ" target="_blank" title="https://youtu.be/akdZefuvBAQ"><img src="https://i.ytimg.com/vi/akdZefuvBAQ/default.jpg"></a></div>



<a name="164927810"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164927810" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164927810">(May 05 2019 at 17:10)</a>:</h4>
<p>I blindly goto definition of a field, and it seems to find one inside a real-world syn-macro! (the text range is wrong, but that shouldn't be terribly hard to fix)</p>



<a name="164927811"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164927811" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164927811">(May 05 2019 at 17:10)</a>:</h4>
<p>and goto definition works for macros by example...</p>



<a name="164927812"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164927812" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164927812">(May 05 2019 at 17:10)</a>:</h4>
<p>wow!</p>



<a name="164930131"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164930131" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164930131">(May 05 2019 at 18:18)</a>:</h4>
<p>really, working with syn is <strong>so</strong> much more pleasant now</p>



<a name="164999878"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164999878" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164999878">(May 06 2019 at 17:12)</a>:</h4>
<p>Hey, finally i would start implement the new iterator approach of the <code>TokenSource</code> trait. (But i will be vacation this week, but let discuss an issues i don't know how to solve first. )<br>
One of the goals of the new api is to simplify tree traversal of subtree (The current <code>SubtreeSource</code> and <code>SubtreeWalker</code>). For <code>bump</code>, we can just push the current level and move to deeper subtree. The hard part is how do we handle <code>lookahead</code>.</p>
<p>The trivial solution is we just look-ahead current level  tokens, which is how <code>rustc</code> works. And It should work even multiple byte punct and deeper subtree. If we find that it is a subtree, we just return the delimiter. It is because normally the usage of lookahead is finding what structure is next to current token. On the other hand, this method is very efficient as we only need to do <code>subtree.token_trees[n] </code>.</p>
<p>However, this solution won't work in current implementation of ra  because of the <code>DOLLAR</code>tokens. <br>
Imagine the following codes:</p>
<div class="codehilite"><pre><span></span>foo $               $
     1 + $     $ + 1
          a + b
</pre></div>


<p>Let say the current token is <code>foo</code>, if we want to <code>lookahead(3)</code>, we have to walk the tree step by step. (check it is a subtree, go next level, and again). And it violates the goal to simplify the tree traversal of <code>subtree</code> algorithm.  </p>
<p><span class="user-mention" data-user-id="133169">@matklad</span> Do you have any idea about this problem?</p>



<a name="164999951"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164999951" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164999951">(May 06 2019 at 17:13)</a>:</h4>
<p>Hm, interesting!</p>



<a name="164999968"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/164999968" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#164999968">(May 06 2019 at 17:14)</a>:</h4>
<p>One thing we should do is to take a look at how <code>syn</code> handles parsing</p>



<a name="165000021"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165000021" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165000021">(May 06 2019 at 17:14)</a>:</h4>
<p>our token-tree is very much a mirror of proc_macro2 tt</p>



<a name="165000032"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165000032" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165000032">(May 06 2019 at 17:14)</a>:</h4>
<p>and syn does have some nice API for bounded lookahead</p>



<a name="165000061"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165000061" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165000061">(May 06 2019 at 17:14)</a>:</h4>
<p>I think we need to bound lookahead by some amount</p>



<a name="165000069"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165000069" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165000069">(May 06 2019 at 17:15)</a>:</h4>
<p>that is, get rid of general lookahead(n: usize)</p>



<a name="165000098"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165000098" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165000098">(May 06 2019 at 17:15)</a>:</h4>
<p>and have separate lookahead1, lookahead2, lookahead3</p>



<a name="165000112"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165000112" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165000112">(May 06 2019 at 17:15)</a>:</h4>
<p>We indeed shouldn't descent into trees with non-empty delimiters in lookahead</p>



<a name="165000128"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165000128" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165000128">(May 06 2019 at 17:15)</a>:</h4>
<p>but for dollars it seems like we should do this?</p>



<a name="165000189"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165000189" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165000189">(May 06 2019 at 17:16)</a>:</h4>
<p>and that means that yeah, lookahdead for dollars will need to go deeper</p>



<a name="165000358"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165000358" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165000358">(May 06 2019 at 17:18)</a>:</h4>
<p>Then let me study how proc_macro2 tt works in this week first :)</p>



<a name="165000399"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165000399" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165000399">(May 06 2019 at 17:19)</a>:</h4>
<p>Yeah, I feel like i might need to take a closer look at syn as well</p>



<a name="165000413"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165000413" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165000413">(May 06 2019 at 17:19)</a>:</h4>
<p>the API is just awesome</p>



<a name="165000696"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165000696" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165000696">(May 06 2019 at 17:22)</a>:</h4>
<blockquote>
<p>The API is just awesome</p>
</blockquote>
<p>Yes, and <code>quote</code> too. Sometimes while workings on ra_mbe, i wish i could use <code>quote!</code> to build a tt::subtree :)</p>



<a name="165000870"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165000870" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165000870">(May 06 2019 at 17:24)</a>:</h4>
<p>And FYI, i added macro expansion support in my wasm toy : <a href="https://web-ra-syntax-node.herokuapp.com/" target="_blank" title="https://web-ra-syntax-node.herokuapp.com/">https://web-ra-syntax-node.herokuapp.com/</a></p>



<a name="165001082"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165001082" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165001082">(May 06 2019 at 17:26)</a>:</h4>
<p>Hm, why don't we add it to rust-analyzer proper?</p>



<a name="165001109"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165001109" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165001109">(May 06 2019 at 17:27)</a>:</h4>
<p>I mean we can totally have "expand macro" action/intention</p>



<a name="165001402"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165001402" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165001402">(May 06 2019 at 17:30)</a>:</h4>
<p>Is is possible to create another buffer (like Show SyntaxNode) and display the expanded selected macro invocation?</p>



<a name="165001444"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165001444" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165001444">(May 06 2019 at 17:30)</a>:</h4>
<p>We do this for "show syntax tree" command</p>



<a name="165001628"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165001628" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165001628">(May 06 2019 at 17:32)</a>:</h4>
<p>So it is possible. Okay, let me add this to my TODO list :)</p>



<a name="165001865"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/165001865" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Kolb <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#165001865">(May 06 2019 at 17:35)</a>:</h4>
<blockquote>
<p>And FYI, i added macro expansion support in my wasm toy : <a href="https://web-ra-syntax-node.herokuapp.com/" target="_blank" title="https://web-ra-syntax-node.herokuapp.com/">https://web-ra-syntax-node.herokuapp.com/</a></p>
</blockquote>
<p>Oh that is neat!</p>



<a name="166140882"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166140882" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166140882">(May 21 2019 at 04:14)</a>:</h4>
<p>Here is my study about how <code>syn</code> crate handle the tree-traversal problem:</p>
<ul>
<li>It just ignore Delimiter::None. <a href="https://github.com/dtolnay/syn/blob/b9cd1bbcf457b036d11f713abcb7b7fdfa09d3cb/src/buffer.rs#L210" target="_blank" title="https://github.com/dtolnay/syn/blob/b9cd1bbcf457b036d11f713abcb7b7fdfa09d3cb/src/buffer.rs#L210">code</a></li>
<li>The "meat" of the <code>TokenStream</code> is <a href="https://github.com/dtolnay/syn/blob/b9cd1bbcf457b036d11f713abcb7b7fdfa09d3cb/src/buffer.rs#L25-L46" target="_blank" title="https://github.com/dtolnay/syn/blob/b9cd1bbcf457b036d11f713abcb7b7fdfa09d3cb/src/buffer.rs#L25-L46"><code>TokenBuffer</code></a> which can create a cheaply copyable cursor and can handle tree-traversal and look-ahead cheaply. However, the whole original <code>TokenTree</code> has to be converted to <code>TokenBuffer</code> first. I don't know whether we want to do it. </li>
<li>It is a very rough idea:  how about we store the TokenStream in salsa instead of TokenTree ? What i mean is, instead of <code>ast_to_token_tree</code>in <code>syntax_bridge.rs</code>, we use <code>ast_to_token_stream</code>, and replace all <code>TokenTree</code> in macro related code to <code>hir</code> to use<code>TokenStream</code>.</li>
</ul>



<a name="166148725"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166148725" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166148725">(May 21 2019 at 07:28)</a>:</h4>
<p>Hm, TokenStream is basically a TokenTree::Subtree with delimeters = None?</p>



<a name="166148751"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166148751" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166148751">(May 21 2019 at 07:29)</a>:</h4>
<p>There's a feeling that <code>TokenStream</code> is a historical accident, which is not really required: <a href="https://github.com/rust-lang/rust/pull/49597#issuecomment-378409251" target="_blank" title="https://github.com/rust-lang/rust/pull/49597#issuecomment-378409251">https://github.com/rust-lang/rust/pull/49597#issuecomment-378409251</a></p>



<a name="166149250"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166149250" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166149250">(May 21 2019 at 07:39)</a>:</h4>
<blockquote>
<p>Hm, TokenStream is basically a TokenTree::Subtree with delimeters = None?</p>
</blockquote>
<p>I mean the proc_macro2 TokenStream, which you can think of a stream of tokens.</p>



<a name="166149336"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166149336" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166149336">(May 21 2019 at 07:41)</a>:</h4>
<p>Or we can name it as <code>TokenBuffer</code></p>



<a name="166149483"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166149483" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166149483">(May 21 2019 at 07:43)</a>:</h4>
<p>Yeah, this is what I am talking about</p>



<a name="166149517"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166149517" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166149517">(May 21 2019 at 07:44)</a>:</h4>
<p>TokenStream is <em>just</em> a Vec of tokens, or Group without delimiters: <a href="https://github.com/alexcrichton/proc-macro2/blob/373fab08aefccdd76afa29798def239b67783881/src/fallback.rs#L19-L21" target="_blank" title="https://github.com/alexcrichton/proc-macro2/blob/373fab08aefccdd76afa29798def239b67783881/src/fallback.rs#L19-L21">https://github.com/alexcrichton/proc-macro2/blob/373fab08aefccdd76afa29798def239b67783881/src/fallback.rs#L19-L21</a></p>



<a name="166149544"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166149544" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166149544">(May 21 2019 at 07:44)</a>:</h4>
<p>so, it's not an essential type for the API</p>



<a name="166150669"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166150669" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166150669">(May 21 2019 at 08:04)</a>:</h4>
<p>Sorry, it is my fault to express my idea not very well (My English is not good).   What i really mean is, we use <a href="https://github.com/dtolnay/syn/blob/6533607f91686545cb034d2838beea338d9d0742/src/buffer.rs#L41" target="_blank" title="https://github.com/dtolnay/syn/blob/6533607f91686545cb034d2838beea338d9d0742/src/buffer.rs#L41"><code>TokenBuffer</code></a> to replace the <code>tt:TokenTree</code> stored in <code>hir</code> and named it as <code>TokenStream</code>. Although it is not an essential type for the API, but it allow us to prevent the conversion back and forth.  </p>
<p>The original reason we have <code>SubtreeWalker</code> , is to prevent the whole tree conversion but i start to believe we should do the conversion anyway.</p>



<a name="166151356"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166151356" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166151356">(May 21 2019 at 08:13)</a>:</h4>
<p>Ah, yeah, that seems resonable!</p>



<a name="166151376"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166151376" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166151376">(May 21 2019 at 08:13)</a>:</h4>
<p>Though, my understanding is that TokenBuffer uses unsafe heavily, for performance</p>



<a name="166151432"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166151432" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166151432">(May 21 2019 at 08:14)</a>:</h4>
<p>I wonder if we should start with slower, but safe impl instead?</p>



<a name="166151446"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166151446" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166151446">(May 21 2019 at 08:14)</a>:</h4>
<p>That is, can we replicate API of <code>TokenBuffer</code> in safe code?</p>



<a name="166157624"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166157624" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166157624">(May 21 2019 at 09:51)</a>:</h4>
<p>I don't know  :) , but maybe i will try to make some prototype first.</p>
<p>OTOH, what is your opinion about the priorities of following things about macros ?<br>
1. Improve name resolutions of mbe, mainly introduce proper macro scopes for local macros (support macro_use)<br>
2. Proper handling different kinds of macro invocations (stmts, expr, pat)<br>
3. NavigationTarget api <br>
4. <code>TokenSource</code> api improvement<br>
5. Start hacking proc-macro</p>



<a name="166158186"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166158186" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166158186">(May 21 2019 at 09:59)</a>:</h4>
<p>I would like the following priority:</p>
<ul>
<li>refactoring, simplifying and future-proofing existing infrastructrue: long-term maintainence is always a good goal!</li>
<li>making goto defeinition use the correct ranges (I feel this could be the first step towards supporting hygiene)</li>
<li>making hygiene proof-of-concept</li>
<li>making proc-macro proof-of-concept</li>
<li>improving "language coverage" (ie. expand more macros, resolve more names, etc)</li>
</ul>



<a name="166159795"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166159795" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166159795">(May 21 2019 at 10:25)</a>:</h4>
<p>Okay :)</p>



<a name="166159946"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166159946" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166159946">(May 21 2019 at 10:27)</a>:</h4>
<p>To give a broader picture, the current goal of RLS-2.0 is to see what things are possible and not to make a perfect IDE. Experimenting with stuff like hygiene is better aligned with this goal than just making everything correct 100% :-)</p>



<a name="166377386"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166377386" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166377386">(May 23 2019 at 16:33)</a>:</h4>
<div class="codehilite"><pre><span></span><span class="sd">/// `TokenSource` abstracts the source of the tokens parser operates one.</span>
<span class="sd">///</span>
<span class="sd">/// Hopefully this will allow us to treat text and token trees in the same way!</span>
<span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">TokenSource</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="k">type</span> <span class="nc">Cursor</span>: <span class="nc">TokenCursor</span><span class="p">;</span><span class="w"></span>

<span class="w">    </span><span class="k">fn</span> <span class="nf">current</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">TokenCursor</span><span class="p">;</span><span class="w"></span>

<span class="w">    </span><span class="sd">/// Lookahead 1 token</span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">Lookahead1</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">TokenCursor</span><span class="p">;</span><span class="w"></span>
<span class="w">    </span><span class="sd">/// Lookahead 2 tokens</span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">Lookahead2</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">TokenCursor</span><span class="p">;</span><span class="w"></span>
<span class="w">    </span><span class="sd">/// Lookahead 3 tokens</span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">Lookahead3</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">TokenCursor</span><span class="p">;</span><span class="w"></span>

<span class="w">    </span><span class="sd">/// bump cursor to next token</span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">bump</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>

<span class="sd">/// `TokenCursor` abstracts the cursor of `TokenSource` operates one.</span>
<span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">TokenCursor</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="sd">/// What is the current token?</span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">token_kind</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">SyntaxKind</span><span class="p">;</span><span class="w"></span>

<span class="w">    </span><span class="sd">/// Is the current token joined to the next one (`&gt; &gt;` vs `&gt;&gt;`).</span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">is_token_joint_to_next</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="kt">bool</span><span class="p">;</span><span class="w"></span>

<span class="w">    </span><span class="sd">/// Is the current token a specified keyword?</span>
<span class="w">    </span><span class="k">fn</span> <span class="nf">is_keyword</span><span class="p">(</span><span class="o">&amp;</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">kw</span>: <span class="kp">&amp;</span><span class="kt">str</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="kt">bool</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>


<p>Do you think this new interface reasonable ? <span class="user-mention" data-user-id="133169">@matklad</span> <br>
I remember you said you want to use <code>is_composite</code> instead of <code>is_token_joint_to_next</code>. However, it will push back the responsibility of recognizing composite tokens to <code>mbe</code> and text stream. I don't know whether it is good approach.</p>



<a name="166377717"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166377717" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166377717">(May 23 2019 at 16:36)</a>:</h4>
<p>Seems good to me!</p>



<a name="166377755"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166377755" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166377755">(May 23 2019 at 16:37)</a>:</h4>
<p><code>TokenCursor</code> might be named better, it's not really a courrsor, because it doesn't move. Perhaps just <code>Token</code>?</p>



<a name="166377821"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166377821" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166377821">(May 23 2019 at 16:38)</a>:</h4>
<p>I think <code>TokenCursor</code> might contain references, so it should be paramterized over a lifetime</p>



<a name="166378058"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166378058" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166378058">(May 23 2019 at 16:41)</a>:</h4>
<p>Yeah, <code>Token</code> is much better name and that should be paramteriaed too. <span aria-label="+1" class="emoji emoji-1f44d" role="img" title="+1">:+1:</span></p>



<a name="166385831"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166385831" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Kolb <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166385831">(May 23 2019 at 18:17)</a>:</h4>
<p>maybe <code>is_token_joined_to_next</code>?</p>



<a name="166469710"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166469710" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166469710">(May 24 2019 at 16:49)</a>:</h4>
<p><span class="user-mention" data-user-id="203366">@Jeremy Kolb</span> Sure</p>



<a name="166469829"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166469829" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166469829">(May 24 2019 at 16:50)</a>:</h4>
<p><span class="user-mention" data-user-id="133169">@matklad</span> Do you think i should change the <code>Parser</code> to use <code>lookahead_1</code> instead of <code>nth</code> too ?</p>



<a name="166469875"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166469875" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166469875">(May 24 2019 at 16:51)</a>:</h4>
<p>Actually, I think it's better to just live nth, but panic if n is too large</p>



<a name="166470332"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166470332" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166470332">(May 24 2019 at 16:57)</a>:</h4>
<p>Okay, and i faced another problem to change the TokenSource interface. The associated item of new TokenSource trait make every parser code to have to include the generic type argument. For example:</p>
<div class="codehilite"><pre><span></span><span class="k">fn</span> <span class="nf">lhs</span><span class="p">(</span><span class="n">p</span>: <span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="n">Parser</span><span class="p">)</span><span class="w"></span>
</pre></div>


<p>became:</p>
<div class="codehilite"><pre><span></span><span class="k">fn</span> <span class="nf">lhs</span><span class="o">&lt;</span><span class="n">TT</span><span class="o">&gt;</span><span class="p">(</span><span class="w"> </span><span class="n">p</span>: <span class="kp">&amp;</span><span class="nc">mut</span><span class="w"> </span><span class="n">Parser</span><span class="o">&lt;</span><span class="n">TT</span><span class="o">&gt;</span><span class="p">)</span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">T</span><span class="w"> </span>: <span class="nc">Token</span><span class="w"></span>
</pre></div>


<p>I think i will be change it back to return (At least temp):</p>
<div class="codehilite"><pre><span></span>struct Token {
  token_kind : SyntaxKind,
  is_token_jointed_to_next: bool,
}
</pre></div>



<a name="166471358"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/166471358" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#166471358">(May 24 2019 at 17:10)</a>:</h4>
<p>yeah, it seems like we want concrete type there!</p>



<a name="167105559"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167105559" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167105559">(Jun 02 2019 at 05:10)</a>:</h4>
<p>I want to work on the following item: </p>
<blockquote>
<p>making goto defeinition use the correct ranges (I feel this could be the first step towards supporting hygiene)</p>
</blockquote>
<p>If i understand correctly, we should start from refactoring "NavTarget" from the following comment in github, right ?</p>
<blockquote>
<p>review NavigationTarget and other stuff in display to make sure it doesn't try to do semantic analysis (this is not really macro-related)</p>
</blockquote>
<p>But i am a little bit  lost, what do you mean about semantic analysis ? Would you mind to guide me what should i start with ?</p>



<a name="167113889"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167113889" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167113889">(Jun 02 2019 at 09:22)</a>:</h4>
<p>sure, let me take a look...</p>



<a name="167113957"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167113957" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167113957">(Jun 02 2019 at 09:24)</a>:</h4>
<p>The problem are <code>node,</code> <code>docs</code> and <code>description</code> methods</p>



<a name="167113969"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167113969" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167113969">(Jun 02 2019 at 09:25)</a>:</h4>
<p>NavTarget intended to be a simple POD type to communicate with LSP, so it shouldn't really have any semantic meaning</p>



<a name="167113970"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167113970" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167113970">(Jun 02 2019 at 09:25)</a>:</h4>
<p>The problem with these three methods is that they take <code>&amp;self</code>, <code>db</code> and do some computation</p>



<a name="167114018"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167114018" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167114018">(Jun 02 2019 at 09:26)</a>:</h4>
<p>But they can't really relibly to extract doc strings, etc</p>



<a name="167114032"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167114032" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167114032">(Jun 02 2019 at 09:27)</a>:</h4>
<p>what we should do is to add <code>docs</code> and <code>descripiton</code> as <strong>fields</strong> to NavTarget, and fill them in during construction, when we have much better understand what is the hir entity we construct a nav target for</p>



<a name="167114166"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167114166" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167114166">(Jun 02 2019 at 09:31)</a>:</h4>
<p>And the reason is why this is related to macros is that <code>NavTarget::node</code> simply doesn't work if there are macros: it purposefully knows only range in the original file,  it can't featch the node from macro expansion</p>



<a name="167821747"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167821747" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167821747">(Jun 11 2019 at 06:03)</a>:</h4>
<p>As <a href="https://github.com/rust-analyzer/rust-analyzer/pull/1388" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/pull/1388">#1388</a> is landed, what is the next step ? Remove <code>HirId::as_original_file</code> ?</p>



<a name="167828733"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167828733" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167828733">(Jun 11 2019 at 08:11)</a>:</h4>
<p>I think that would be the last step</p>



<a name="167828801"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167828801" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167828801">(Jun 11 2019 at 08:12)</a>:</h4>
<p>first, we need to provide an alternative, and only then remove the old way (migrating <strong>all</strong> ide_api to it, not only goto def)</p>



<a name="167828810"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167828810" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167828810">(Jun 11 2019 at 08:12)</a>:</h4>
<p>Let's talk through what needs to be done here!</p>



<a name="167829007"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167829007" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167829007">(Jun 11 2019 at 08:15)</a>:</h4>
<p>Let's say we invoke <code>goto_defeinition</code> and get a <code>Def</code> back, here: <a href="https://github.com/rust-analyzer/rust-analyzer/blob/3f5f9f0560fa662e770b607f05ec4881e4d011c5/crates/ra_ide_api/src/goto_definition.rs#L66" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/blob/3f5f9f0560fa662e770b607f05ec4881e4d011c5/crates/ra_ide_api/src/goto_definition.rs#L66">https://github.com/rust-analyzer/rust-analyzer/blob/3f5f9f0560fa662e770b607f05ec4881e4d011c5/crates/ra_ide_api/src/goto_definition.rs#L66</a></p>



<a name="167829081"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167829081" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167829081">(Jun 11 2019 at 08:16)</a>:</h4>
<p>That <code>Def</code> might originate in macro expansion, so we can't literally naviagate to it. Rather, we should navigate to the defining macro</p>



<a name="167829139"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167829139" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167829139">(Jun 11 2019 at 08:17)</a>:</h4>
<p>That means the trick is setting <code>full_range</code>, <code>focus_range</code> and <code>file_id</code> correctly: <a href="https://github.com/rust-analyzer/rust-analyzer/blob/3f5f9f0560fa662e770b607f05ec4881e4d011c5/crates/ra_ide_api/src/display/navigation_target.rs#L23-L24" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/blob/3f5f9f0560fa662e770b607f05ec4881e4d011c5/crates/ra_ide_api/src/display/navigation_target.rs#L23-L24">https://github.com/rust-analyzer/rust-analyzer/blob/3f5f9f0560fa662e770b607f05ec4881e4d011c5/crates/ra_ide_api/src/display/navigation_target.rs#L23-L24</a></p>



<a name="167829474"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167829474" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167829474">(Jun 11 2019 at 08:22)</a>:</h4>
<p>The behavior we want here is that we should try to set ranges correctly, falling back to the range of the whole macro invocation if that fails</p>



<a name="167829479"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167829479" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167829479">(Jun 11 2019 at 08:22)</a>:</h4>
<p>let me give an example</p>



<a name="167829521"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167829521" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167829521">(Jun 11 2019 at 08:23)</a>:</h4>
<p>Let's say, we have</p>
<div class="codehilite"><pre><span></span>macro_rules! define_fn {
   ($name:ident) =&gt; (fn $name() {})
}

define_fn!(
    foo
)

fn main() {
   foo() // &lt;- user presses goto definition here
}
</pre></div>



<a name="167829578"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167829578" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167829578">(Jun 11 2019 at 08:24)</a>:</h4>
<p>In this case, we should arrive at</p>
<div class="codehilite"><pre><span></span>define_fn!(
    &lt;*&gt;foo&lt;*&gt; // &lt;- this range
)
</pre></div>



<a name="167830673"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167830673" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167830673">(Jun 11 2019 at 08:42)</a>:</h4>
<p>However, for </p>
<div class="codehilite"><pre><span></span>macro_rules! define_fn {
   ($name:ident) =&gt; (fn $name() {})
}

define_fn!();

fn main() {
   foo()
}
</pre></div>


<p>we should navigate to the whole of <code>define_fn!()</code>, because the identifier is created inside the macro</p>



<a name="167830804"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167830804" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167830804">(Jun 11 2019 at 08:45)</a>:</h4>
<p>So the first step should be add a test for that, and implement <code>fn HirFileId::original_file_and_range(&amp;self, db: &amp;RootDatabase, range: TextRange) -&gt; (HirFileId, TextRange)</code>, right ?</p>



<a name="167830805"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167830805" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167830805">(Jun 11 2019 at 08:45)</a>:</h4>
<p>I think the logic for this should be as follows.</p>
<p>When we expand a macro, we remember a mapping of ranges from macro expansion to a macro call (Vec&lt;(TextRange, TextRange)&gt;)</p>



<a name="167830830"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167830830" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167830830">(Jun 11 2019 at 08:45)</a>:</h4>
<p>yeah, starting with a test is a good idea! Integration test in ra_ide_api would be best</p>



<a name="167830886"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167830886" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167830886">(Jun 11 2019 at 08:46)</a>:</h4>
<p>as for API I am not exactly sure what we need it</p>



<a name="167830914"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167830914" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167830914">(Jun 11 2019 at 08:46)</a>:</h4>
<p><code>original_file_and_range</code> is maybe ok as a high-level API, but I'd also like to think about how low-level API should work</p>



<a name="167830962"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167830962" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167830962">(Jun 11 2019 at 08:47)</a>:</h4>
<p>Sure</p>



<a name="167830982"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167830982" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167830982">(Jun 11 2019 at 08:47)</a>:</h4>
<blockquote>
<p>When we expand a macro, we remember a mapping of ranges from macro expansion to a macro call (Vec&lt;(TextRange, TextRange)&gt;)</p>
</blockquote>
<p>What is the relationship of it  with TokenId ?</p>



<a name="167831300"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167831300" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167831300">(Jun 11 2019 at 08:53)</a>:</h4>
<p>I think we need <code>HirFileId::parent_expansion() -&gt; Option&lt;(HirFileId, ExpansionInfo)&gt;</code>, as one-step expansion, will be better</p>



<a name="167831308"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167831308" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167831308">(Jun 11 2019 at 08:53)</a>:</h4>
<p>the <code>ExpansionInfo</code> will have methods to map ranges in both directions</p>



<a name="167831368"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167831368" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167831368">(Jun 11 2019 at 08:54)</a>:</h4>
<blockquote>
<p>What is the relationship of it with TokenId ?</p>
</blockquote>
<p>So, <code>TokenId</code>s should be a way implement the mapping yeah</p>



<a name="167831383"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167831383" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167831383">(Jun 11 2019 at 08:55)</a>:</h4>
<p>when we expand macro, we get a sort of <code>&amp;Token -&gt; &amp;Token</code> map, and we need to somehow reconstruct the ranges map our of that info</p>



<a name="167831405"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167831405" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167831405">(Jun 11 2019 at 08:55)</a>:</h4>
<p>Seem like we already have the TextRange mapping ?<br>
<a href="https://github.com/rust-analyzer/rust-analyzer/blob/b79e6294a68fd41f0a3dbd9eb907dfe99646d77e/crates/ra_mbe/src/syntax_bridge.rs#L12" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/blob/b79e6294a68fd41f0a3dbd9eb907dfe99646d77e/crates/ra_mbe/src/syntax_bridge.rs#L12">https://github.com/rust-analyzer/rust-analyzer/blob/b79e6294a68fd41f0a3dbd9eb907dfe99646d77e/crates/ra_mbe/src/syntax_bridge.rs#L12</a></p>



<a name="167831409"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167831409" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167831409">(Jun 11 2019 at 08:55)</a>:</h4>
<p>It might be the case that we don't need to reconstruct the actual Vec&lt;TextRange, TextRange&gt; mapping</p>



<a name="167831416"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167831416" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167831416">(Jun 11 2019 at 08:55)</a>:</h4>
<p>Yeah, <code>TokenMap</code> is a bit of that infra</p>



<a name="167831461"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167831461" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167831461">(Jun 11 2019 at 08:56)</a>:</h4>
<p>but it never was finished :-)</p>



<a name="167831521"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167831521" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167831521">(Jun 11 2019 at 08:57)</a>:</h4>
<p>How about that: Let me combined all information here, and <strong>TRY</strong> to make an implementation to pass that test I mentioned before, and then discuss it in that PR ?</p>



<a name="167831578"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167831578" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167831578">(Jun 11 2019 at 08:58)</a>:</h4>
<p>SGTM!</p>



<a name="167831591"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167831591" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167831591">(Jun 11 2019 at 08:58)</a>:</h4>
<p>Hm, now that I think about it, we <em>used</em> to have this working some time ago</p>



<a name="167831598"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167831598" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167831598">(Jun 11 2019 at 08:58)</a>:</h4>
<p>but then I've removed it, when we went to a proper macro expansion... Let me dig out some old codes...</p>



<a name="167832212"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167832212" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167832212">(Jun 11 2019 at 09:08)</a>:</h4>
<p><a href="https://github.com/rust-analyzer/rust-analyzer/commit/5299a35e3dc484ea2e7d42cfeed89aee806425d3" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/commit/5299a35e3dc484ea2e7d42cfeed89aee806425d3">https://github.com/rust-analyzer/rust-analyzer/commit/5299a35e3dc484ea2e7d42cfeed89aee806425d3</a></p>



<a name="167832216"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167832216" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167832216">(Jun 11 2019 at 09:08)</a>:</h4>
<p>so this is a bit from that old implementation</p>



<a name="167832221"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167832221" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167832221">(Jun 11 2019 at 09:09)</a>:</h4>
<p>it handles extend selection</p>



<a name="167832242"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167832242" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167832242">(Jun 11 2019 at 09:09)</a>:</h4>
<p>and is probably the most interesting case for macro expanstion</p>



<a name="167832310"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167832310" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167832310">(Jun 11 2019 at 09:10)</a>:</h4>
<p>it first takes original ranges, transforms it into a range inside the macro, runs expand selection in the macro, and maps the resulting range back</p>



<a name="167832318"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167832318" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167832318">(Jun 11 2019 at 09:10)</a>:</h4>
<p>so, that's why you need bidirectional mapping</p>



<a name="167832824"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167832824" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167832824">(Jun 11 2019 at 09:18)</a>:</h4>
<p>Um.. interesting. So we need something like this which replace TokenMap ? <br>
<a href="https://github.com/rust-analyzer/rust-analyzer/blob/792899587647f5aa0293c2588173677682187c0a/crates/ra_analysis/src/macros.rs#L44" target="_blank" title="https://github.com/rust-analyzer/rust-analyzer/blob/792899587647f5aa0293c2588173677682187c0a/crates/ra_analysis/src/macros.rs#L44">https://github.com/rust-analyzer/rust-analyzer/blob/792899587647f5aa0293c2588173677682187c0a/crates/ra_analysis/src/macros.rs#L44</a></p>



<a name="167832879"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167832879" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167832879">(Jun 11 2019 at 09:18)</a>:</h4>
<p>Or extend TokenMap.</p>



<a name="167833353"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/MBE%20discussion/near/167833353" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/MBE.20discussion.html#167833353">(Jun 11 2019 at 09:25)</a>:</h4>
<p>yeah, i think we need to extend token map. Or maybe build a ranges map from token map</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>